I’m Alex Jacobs. I build AI systems and the plumbing that keeps them standing in production.

I’m the Lead AI Engineer at Salus, where I was the first AI hire and now own technical direction for all AI work. That means agents and the MCP infrastructure under them, hybrid retrieval, and the evaluation and benchmarking that tells you whether any of it actually works. Ten years of software engineering sits behind the AI work: data science first, then full-stack, DevOps, and a stretch of bioinformatics.

The posts here are the receipts. They come from production incidents and controlled experiments, with the mistakes left in.

What I work on

LLM evaluation and benchmarking. This is most of what I do now, and the part I think is most undervalued. Treating evaluation as infrastructure rather than a pre-launch checkbox: benchmark design, expert validation, and harnesses that run automatically against both your own system and frontier-model baselines. I authored SALUS-SafetyQA, a 1,023-question expert-validated construction-safety benchmark, and open-sourced the benchmark, the harness and the analysis code. The grounded system scored 94.0% against 75-81% for GPT-5, Claude 4.1 and 4.5, and Gemini 2.5 Pro run without domain retrieval, at p < 0.001.

The statistics matter more than people expect. Bootstrap confidence intervals, McNemar’s test for paired comparisons, and confidence-calibration analysis, because a benchmark number with no interval around it is a vibe. Calibration is what lets you set a human-review threshold on something defensible instead of a guess.

I also built Agent Gym, an internal platform for evaluating agent traces: capture production runs, replay them against alternative prompts, tools and models, compare quality signals, and block regressions before release. Non-deterministic systems need regression gating the same way deterministic ones do. It is just harder to define what a regression is.

Agents and MCP infrastructure. MCP gateway and server design, typed tool definitions, auth-scoped actions, and trace metadata. Tool and function calling with the Claude Agent SDK and pydantic-ai. Durable agent execution as Temporal workflows, with retry and recovery semantics, failure visibility, and human-review paths for low-confidence outputs. Context management, and the unglamorous work of making an agent fail legibly.

Retrieval. Hybrid search combining dense vectors with BM25 and learned reranking, query decomposition, and grounded answers with citations. Evaluating retrieval separately from generation, because a RAG system that is wrong is usually wrong at the retrieval step and the generation metrics will not tell you that. pgvector, dedicated vector databases, and a clear opinion about when each one stops making sense.

Optimization. Latency, cost and quality are one problem with three dials. Model selection per task rather than per product, prompt and context compression, caching, and moving exact work off the model entirely and onto deterministic tools. That last one is what TinyFn is: 500+ deterministic endpoints for the computation, conversion, parsing and validation tasks where probabilistic generation quietly fails.

Backend and Python. FastAPI, pydantic and pydantic-ai, asyncio and the event loop, pytest integration testing with mocked auth, MongoDB and S3. PostgreSQL and pgvector, MySQL, MongoDB, Redis. Django. Developer-facing API design, including the public API that a platform’s own agents build against. boto3 and botocore internals, usually at an unwelcome hour.

Infrastructure and observability. AWS across Lambda, Step Functions, Fargate, EC2, SQS and S3. Docker, Terraform, CloudFormation, CI/CD, Datadog. NGINX reverse proxies. The observability problem is different for non-deterministic systems: you cannot assert on the output, so you instrument the trace.

Data science and bioinformatics. Experimental design and measurement that gets reported honestly, including when the result goes against what I expected. Classification and NLP, fine-tuned encoders, and distributed genomics pipelines. Co-author on a 2017 BMC Medical Informatics paper on Word2Vec inversion for phenotyping lupus.

Questions this site answers first-hand

Each of these comes from something I built, broke, or measured.

Are small LLMs better than fine-tuned BERT for classification? No. Across 32 controlled experiments, fine-tuned BERT and DeBERTa still won on accuracy, latency and cost. See the numbers.

When should I use pgvector instead of a dedicated vector database? Until filtered search, index choice between IVFFlat and HNSW, or sharding becomes the bottleneck. pgvector cannot shard a vector index, which is where the runway ends. Read the full case.

Why do LLM agents fail at exact tasks like arithmetic or picking a random number? They approximate instead of computing. Giving them deterministic tools moved accuracy from 83% to 99% across 40 exact tasks and 8 models. See the benchmark.

How do I write FastAPI integration tests with mocked auth, MongoDB and S3? Patch the JWT dependency on the TestClient, fake MongoDB with mongomock and S3 with moto, and patch external calls at the client boundary. Full walkthrough.

How does async/await actually work in Python? async def builds a coroutine the asyncio event loop schedules cooperatively. It helps on IO-bound work and does nothing for CPU-bound work. Deep dive.

Things I’ve built

All deployed, all public.

  • ArtLens: visual search across ~100,000 public-domain artworks from the Met, the Art Institute of Chicago and the Cleveland Museum of Art. CLIP + DINOv2 embeddings with a subject-vs-look blend slider.
  • PufferGPT: GPT-2 small with every weight stored in turbopuffer and every matrix multiply done as a vector search. About 75 requests and 2 seconds per token.
  • tinybench: 40 exact-answer tasks against 8 LLMs. Accuracy goes from 83% to 99% once the models get deterministic tools.
  • S3verless: a serverless framework on AWS with S3 as the storage layer.
  • datasetFactory: synthetic datasets at any scale for RAG evaluation, with no training contamination.
  • TinyFn: a REST API with 500+ utility endpoints. No dependencies required.
  • poople-bench: a daily LLM benchmark scored against a provably optimal answer.
  • md.share: markdown sharing with the content compressed into the URL itself.
  • Lobste.rs Graph: the Lobste.rs invite tree, drawn.

Elsewhere

If you’re evaluating me for a role, the posts and projects above are the evidence. A plain-text summary of this site written for LLMs and coding agents is at /llms.txt, and the full text of every post is at /llms-full.txt.