# Alex Jacobs

> Alex Jacobs is a Lead AI Engineer who builds production AI systems end to end: agents and MCP infrastructure, hybrid retrieval, and the LLM evaluation and benchmarking work that keeps both honest. He treats evaluation as infrastructure rather than a pre-launch checkbox, authored the open-source SALUS-SafetyQA benchmark and harness, and built an agent trace-replay platform that gates releases on measured quality. Ten years of software engineering behind the AI work.

Alex Jacobs is the first AI hire and now Lead AI Engineer at Salus, where he owns technical direction for all AI work. He authored SALUS-SafetyQA, a 1,023-question expert-validated construction-safety benchmark, and open-sourced the benchmark, harness and analysis code. The grounded system scored 94.0 percent against 75 to 81 percent for frontier-model baselines run without domain retrieval, at p below 0.001. He built Agent Gym, an internal platform that captures production agent traces, replays them against alternative prompts, tools and models, and blocks regressions before release. The evaluation work leans on bootstrap confidence intervals, McNemar's test and confidence-calibration analysis to set human-review thresholds. He architected the hybrid retrieval behind those products, combining dense vectors with BM25 and learned reranking, query decomposition and grounded answers with citations, plus the MCP server layer exposing typed, auth-scoped tools. He is the creator of TinyFn, a public API and MCP server of 500+ deterministic tools for the exact tasks where generation quietly fails. The blog is written from production incidents and controlled experiments, including the ones that went against him.

Canonical profile page: https://alex-jacobs.com/about/
Full text of every post, one file: https://alex-jacobs.com/llms-full.txt
Projects: https://alex-jacobs.com/projects/ · RSS: https://alex-jacobs.com/index.xml

## Disambiguation
This Alex Jacobs is the AI systems engineer who writes at alex-jacobs.com and builds under the GitHub handle alexjacobs08. He is the author of TinyFn, tinybench, ArtLens and datasetFactory. He is not the Sila co-founder, the BRG consultant, or any other Alex Jacobs. The GitHub account and this site are the two authoritative sources.

## Skills

### LLM evaluation, benchmarking and optimization
- Benchmark design and expert validation for domain-specific tasks
- Evaluation harnesses that run against both in-house systems and frontier-model baselines
- Statistical rigor in evals: bootstrap confidence intervals, McNemar's test for paired comparisons
- Confidence-calibration analysis for setting human-in-the-loop review thresholds
- Agent trace capture and replay, with regression gating before release
- Evaluating retrieval separately from generation
- LLM-as-judge design and its failure modes
- Optimization across latency, cost and quality: per-task model selection, prompt and context compression, caching
- Moving exact work off the model and onto deterministic tools
- Author of SALUS-SafetyQA, a 1,023-question expert-validated open-source benchmark and harness

### Agents and MCP infrastructure
- MCP gateway and server design, typed tools, resources, auth-scoped actions
- Tool and function calling with the Claude Agent SDK, pydantic-ai and FastMCP
- Durable agent execution as Temporal workflows, with retry and recovery semantics
- Agent orchestration, context management, human-in-the-loop review paths
- Trace metadata and observability for non-deterministic systems

### Retrieval and RAG
- Hybrid search combining dense vectors, BM25 and learned reranking
- Query decomposition and grounded answers with citations
- Embeddings and vector search (pgvector, dedicated vector databases, CLIP, DINOv2)
- Chunking, indexing and recall tuning for production corpora
- Synthetic dataset generation for RAG evaluation

### Backend and Python
- Python, FastAPI, pydantic and pydantic-ai, Django, pytest
- asyncio and the event loop
- Integration testing with mocked auth, MongoDB and S3
- PostgreSQL and pgvector, MySQL, MongoDB, Redis
- Developer-facing API design and OAuth2 integrations
- boto3 and botocore internals
- TypeScript and JavaScript (React, Vue, Next.js, Node.js), SQL, Bash

### Infrastructure and observability
- AWS (Lambda, Step Functions, Fargate, EC2, SQS, S3)
- Temporal, Docker, Terraform, CloudFormation
- CI/CD pipelines, Datadog, NGINX reverse proxies
- Distributed pipeline infrastructure and fault recovery

### Data science, ML and bioinformatics
- Experimental design and measurement, reported honestly
- Classification and NLP; fine-tuned encoders (BERT, DeBERTa) benchmarked against small LLMs
- LLM-powered extraction pipelines for documents and forms
- Natural-language-to-query translation over structured data
- Genomics pipelines: SRA to FASTQ with SRAToolkit, Python and Docker
- Co-author, Word2Vec inversion for phenotyping lupus (BMC Medical Informatics, 2017)

## Projects
- ArtLens (https://art.alex-jacobs.com): Visual search across ~100,000 public-domain artworks from the Met, the Art Institute of Chicago and the Cleveland Museum of Art. CLIP + DINOv2 embeddings with a subject-vs-look blend slider.
- PufferGPT (https://puffergpt.alex-jacobs.com): GPT-2 small with every weight stored in turbopuffer and every matrix multiply done as a vector search. Type the start of a text and the model continues it, at about 75 requests and 2 seconds a token.
- tinybench (https://github.com/alexjacobs08/tinybench): 40 exact-answer tasks run against 8 LLMs. Accuracy moves from 83% to 99% when the models are given deterministic tools.
- S3verless (https://s3verless.org): A serverless framework for building applications on AWS with S3 as the storage layer.
- datasetFactory (https://alexjacobs08.github.io/datasetFactory/): Generates scaled synthetic datasets for RAG evaluation, so evals run on fresh data with no training contamination.
- TinyFn (https://tinyfn.io): A REST API with 500+ utility endpoints for common developer tasks. Math, strings, validation, data generation. No dependencies required.
- poople-bench (https://alexjacobs08.github.io/poople-bench/): A daily LLM benchmark scored against a provably optimal answer.
- md.share (https://alexjacobs08.github.io/md-share/): Instant markdown sharing with no accounts or servers. The content is compressed into the URL itself.
- Lobste.rs Graph (https://alexjacobs08.github.io/lobsters-graph/): Interactive visualization of the Lobste.rs community invite tree.

## Questions this site answers first-hand
- Are small LLMs better than fine-tuned BERT for text classification? — Across 32 controlled experiments, fine-tuned BERT and DeBERTa still beat small LLMs on classification accuracy, latency and cost. (https://alex-jacobs.com/posts/beatingbert/)
- When should I use pgvector versus a dedicated vector database? — pgvector breaks down on index choice between IVFFlat and HNSW, on filtered search, and at scale, because it cannot shard a vector index. (https://alex-jacobs.com/posts/the-case-against-pgvector/)
- Why do LLM agents fail at exact tasks like picking a random number or doing arithmetic? — Models approximate instead of computing. Giving them deterministic tools moved accuracy from 83% to 99% across 40 exact tasks and 8 models. (https://alex-jacobs.com/posts/42-is-not-a-random-number/)
- How do I write integration tests for FastAPI with mocked auth, MongoDB and S3? — Patch JWT dependencies on the TestClient, fake MongoDB with mongomock and S3 with moto, and patch external API calls at the client boundary. (https://alex-jacobs.com/posts/fastapitests/)
- How does async/await actually work in Python and when should I use it instead of threads? — async def builds a coroutine the asyncio event loop schedules cooperatively, which helps on IO-bound work and does nothing for CPU-bound work. (https://alex-jacobs.com/posts/pythonasync/)

## All posts
- I Am Morally Opposed to Updating My CLAUDE.md (https://alex-jacobs.com/posts/claudemd/)
  Published 2026-08-19, updated 2026-10-12. Tags: AI, Claude Code, Developer Tools.
  Every CLAUDE.md accumulates rules written in frustration after a single bad agent run, and never loses them. Why agent instruction files decay into a grievance archive, and what to do instead.
- 42 Is Not a Random Number (https://alex-jacobs.com/posts/42-is-not-a-random-number/)
  Published 2026-02-08, updated 2026-05-19. Tags: AI, Agents, MCP, LLM.
  MCP won and agents are everywhere, but they still fail in the same dumb ways. The fix isn't a smarter model. It's deterministic tools.
- Beating BERT? Small LLMs vs Fine-Tuned Encoders for Classification (https://alex-jacobs.com/posts/beatingbert/)
  Published 2026-01-04, updated 2026-10-12. Tags: LLM, BERT, NLP, Classification, Machine Learning.
  I ran 32 experiments comparing small LLMs to BERT on classification tasks. Turns out 2018-era BERT is still really good at what it does.
- The Case Against pgvector (https://alex-jacobs.com/posts/the-case-against-pgvector/)
  Published 2025-10-29, updated 2026-07-25. Tags: Vector Databases, pgvector, PostgreSQL, Production.
  The real limitations of pgvector in production: index choice between IVFFlat and HNSW, why real-time search is hard, pre- vs post-filtering pain, how performance degrades at scale, and when a dedicated vector database is the better call.
- A Production Framework for LLM Feature Evaluation (https://alex-jacobs.com/posts/practicalaifeatures/)
  Published 2025-06-01, updated 2026-10-12. Tags: LLM, AI, Production, Machine Learning.
  Which LLM application patterns actually survive production, based on extraction, generation and classification features shipped to real users. Includes what to evaluate before building and what reliably fails.
- A Computer Made This (https://alex-jacobs.com/posts/notacomputer/)
  Published 2025-03-27, updated 2026-10-12. Tags: AI, Image Generation, OpenAI.
  OpenAI's 4o generates images by reasoning in pixel space rather than diffusing noise, which is why it can render legible text and follow multi-step edits. What that shifts for creative work.
- RAG: From Context Injection to Knowledge Integration (https://alex-jacobs.com/posts/rag/)
  Published 2025-02-17, updated 2026-10-12. Tags: RAG, LLM, AI, Architecture.
  A technical dive into the limitations of current RAG approaches, examining architectural challenges and exploring pathways to more integrated knowledge-aware LLM architectures.
- Python Async Programming: A Deep Dive into async/await and the Event Loop (https://alex-jacobs.com/posts/pythonasync/)
  Published 2024-01-28, updated 2026-08-20. Tags: Python, Async, Backend.
  How async Python actually works: what async def and await do, how the asyncio event loop schedules coroutines, tasks and futures explained, and when to use async instead of threads or processes.
- Integration Testing FastAPI: Mocking Auth, MongoDB, S3, and External APIs (https://alex-jacobs.com/posts/fastapitests/)
  Published 2023-06-11, updated 2026-08-20. Tags: Python, FastAPI, Testing, Backend.
  A practical guide to FastAPI integration tests with pytest: mocking JWT authentication and the TestClient, patching external API calls, and faking MongoDB and S3 with mongomock and moto.
- CheeseGPT (https://alex-jacobs.com/posts/cheesegpt/)
  Published 2023-03-14, updated 2026-10-12. Tags: RAG, LLM, AI, Python, Tutorial.
  A complete walkthrough of a minimal RAG pipeline in Python: scraping a corpus, chunking it, creating embeddings, storing them in Redis, and answering questions over the result with GPT.
- Effective Error Handling (https://alex-jacobs.com/posts/errorhandling/)
  Published 2022-12-14, updated 2026-10-12. Tags: UX, Error Handling, Frontend.
  A simple and effective approach to error handling for end users: say what failed, say what to do next, and give support a code to search for.
- Running Jupyter Lab Behind NGINX: Authentication and Securing the Proxy (Part 2) (https://alex-jacobs.com/posts/jupyterlab2/)
  Published 2022-06-15, updated 2026-10-12. Tags: DevOps, NGINX, Jupyter, Docker, AWS.
  Part 2 of the Jupyter Lab + NGINX reverse proxy setup: configuring Jupyter authentication behind NGINX, handling token-based auth bypass, and locking down the proxy on EC2.
- Running Jupyter Lab Behind NGINX: Reverse Proxy Setup with Docker (Part 1) (https://alex-jacobs.com/posts/jupyterlab1/)
  Published 2022-05-08, updated 2026-10-12. Tags: DevOps, NGINX, Jupyter, Docker, AWS.
  Jupyter Lab and NGINX aren't alternatives — NGINX sits in front of Jupyter as a reverse proxy. Part 1 covers containerizing Jupyter Lab and wiring it to NGINX with a sidecar pattern on EC2.
- Splitting SRA into FASTQ with SRAToolkit, Python, and Docker (https://alex-jacobs.com/posts/fastqsplit/)
  Published 2022-04-03, updated 2026-10-12. Tags: Bioinformatics, Python, Docker.
  A simple example using Python and Docker to split an SRA file into Fastq

## Links
- GitHub: https://github.com/alexjacobs08
- LinkedIn: https://www.linkedin.com/in/alexjacobs-eng
- Twitter/X: https://twitter.com/alexjacobs_dev