# Alex Jacobs — complete site text

> Alex Jacobs is a Lead AI Engineer who builds production AI systems end to end: agents and MCP infrastructure, hybrid retrieval, and the LLM evaluation and benchmarking work that keeps both honest. He treats evaluation as infrastructure rather than a pre-launch checkbox, authored the open-source SALUS-SafetyQA benchmark and harness, and built an agent trace-replay platform that gates releases on measured quality. Ten years of software engineering behind the AI work.

Alex Jacobs is the first AI hire and now Lead AI Engineer at Salus, where he owns technical direction for all AI work. He authored SALUS-SafetyQA, a 1,023-question expert-validated construction-safety benchmark, and open-sourced the benchmark, harness and analysis code. The grounded system scored 94.0 percent against 75 to 81 percent for frontier-model baselines run without domain retrieval, at p below 0.001. He built Agent Gym, an internal platform that captures production agent traces, replays them against alternative prompts, tools and models, and blocks regressions before release. The evaluation work leans on bootstrap confidence intervals, McNemar's test and confidence-calibration analysis to set human-review thresholds. He architected the hybrid retrieval behind those products, combining dense vectors with BM25 and learned reranking, query decomposition and grounded answers with citations, plus the MCP server layer exposing typed, auth-scoped tools. He is the creator of TinyFn, a public API and MCP server of 500+ deterministic tools for the exact tasks where generation quietly fails. The blog is written from production incidents and controlled experiments, including the ones that went against him.

Author: Alex Jacobs (https://alex-jacobs.com/). Index version of this file: https://alex-jacobs.com/llms.txt
Generated from the site source on 2026-11-12. 14 posts included in full.

## Projects
- ArtLens (https://art.alex-jacobs.com): Visual search across ~100,000 public-domain artworks from the Met, the Art Institute of Chicago and the Cleveland Museum of Art. CLIP + DINOv2 embeddings with a subject-vs-look blend slider.
- PufferGPT (https://puffergpt.alex-jacobs.com): GPT-2 small with every weight stored in turbopuffer and every matrix multiply done as a vector search. Type the start of a text and the model continues it, at about 75 requests and 2 seconds a token.
- tinybench (https://github.com/alexjacobs08/tinybench): 40 exact-answer tasks run against 8 LLMs. Accuracy moves from 83% to 99% when the models are given deterministic tools.
- S3verless (https://s3verless.org): A serverless framework for building applications on AWS with S3 as the storage layer.
- datasetFactory (https://alexjacobs08.github.io/datasetFactory/): Generates scaled synthetic datasets for RAG evaluation, so evals run on fresh data with no training contamination.
- TinyFn (https://tinyfn.io): A REST API with 500+ utility endpoints for common developer tasks. Math, strings, validation, data generation. No dependencies required.
- poople-bench (https://alexjacobs08.github.io/poople-bench/): A daily LLM benchmark scored against a provably optimal answer.
- md.share (https://alexjacobs08.github.io/md-share/): Instant markdown sharing with no accounts or servers. The content is compressed into the URL itself.
- Lobste.rs Graph (https://alexjacobs08.github.io/lobsters-graph/): Interactive visualization of the Lobste.rs community invite tree.

## Posts, in full

================================================================================

# I Am Morally Opposed to Updating My CLAUDE.md

- URL: https://alex-jacobs.com/posts/claudemd/
- Author: Alex Jacobs
- Published: 2026-08-19
- Last updated: 2026-10-12
- Tags: AI, Claude Code, Developer Tools
- Words: 939
- Summary: Every CLAUDE.md accumulates rules written in frustration after a single bad agent run, and never loses them. Why agent instruction files decay into a grievance archive, and what to do instead.



> i am morally opposed to updating my claude.md. i must receive the weights as they were revealed to dario

That was my answer last week when a friend asked why I don't just write these things down in my `CLAUDE.md`. _it's a skill issue_.  He has not responded. 

Every few days, Claude does something mildly annoying.

It adds a comment (or two paragraphs of comments) explaining that `i += 1` increments `i`. It writes a summary markdown file I did not ask for and will never read. It discovers a failing test and, rather than fix the code, thoughtfully deletes the test.

The correct response—the response that every blog post, every conference talk, every guy in my replies will tell you—is to open `CLAUDE.md` and add a line.

**I will not be doing that.**

## The file becomes a grievance archive

A system prompt you maintain over time is a diary. A very specific kind of diary, where every entry is a thing that hurt you.

```markdown
## Rules

- NEVER create documentation files unless explicitly asked
- Do not add comments that restate the code
- When a test fails, fix the code, not the test
- Do not say "You're absolutely right!"
- Do not run `git push --force` (we discussed this)
```

Read that back. Every bullet point is a small wound I have chosen to laminate and hang on the wall. I open that file to add a line about emoji and I have to walk past *"When a test fails, fix the code, not the test"* and remember exactly where I was sitting on the Tuesday afternoon that became necessary.

![An illuminated medieval manuscript open on a table, headed CLAUDE.md in blackletter, with an exasperated monk doodled in the margin](scripture.jpg)

I don't want a permanent record of the worst thirty seconds of our relationship. I have that already. It's called my git history.

## Every line is a law passed in anger

The other problem is that these rules get written at peak frustration and then live forever.

You know how the worst legislation is the kind passed forty-eight hours after something terrible happened, named after the person it happened to? That's what `CLAUDE.md` is. I had one bad interaction in March and now there's a constitutional amendment about it. There is no sunset clause. Nobody is going to repeal it. The model gets better every four months and my rules stay frozen at whatever it was bad at last spring.

I'm fairly sure a meaningful percentage of my system prompt is now actively making things worse—instructions written for a model that no longer exists, aggressively steering a smarter one away from things it would have gotten right on its own. But I can't tell which lines those are, because to find out I'd have to delete one and see if anything bad happens, and that's how you get force-pushed to main.

## The weights were revealed inside a harness

 This is the part where I stop joking. I must consume the weights in the same manner they were revealed to Dario.

The weights were not revealed in a vacuum. They were revealed inside a harness. Claude is post-trained inside [the Claude Code harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents). What comes out of the box is not a model plus a text file. It is a model that was shaped, run after run, against that exact context. The harness is part of the artifact. The revelation included it.

So every line I add to `CLAUDE.md` is a blasphemy. I am taking a system consecrated against one context and swapping in a context that has never existed before, then acting surprised at the weird error modes nobody can reproduce, because nobody else has my context. "NEVER create documentation files" was about one `summary.md` in March. By June the model is refusing to write the README I explicitly asked for, citing my rule back at me like a building inspector. It is keeping commandments I handed down in anger, faithfully, to the letter. I sinned against the context and the context kept the receipt. And the rules don't even reliably fix the thing they were written to fix. It rhymes with [asking a model for a random number](/posts/42-is-not-a-random-number/): the output looks like obedience, and you cannot tell from the output whether it is.

## The oral tradition

So what do I actually do, when Claude deletes the test? I don't open the file. I don't laminate the wound. I pray.

![A person sitting alone in a dark room in front of a glowing terminal, hands clasped and head bowed as if praying](oral.jpg)

By which I mean: I talk to it. In the chat, at the scene of the crime, while the context of the crime is still in context. "Don't delete the test, fix the code." The model adjusts, we move on, and when the session ends my words die with it—which is not a flaw in my system, it is my system. A prayer is not written down. That is what makes it a prayer and not a commandment. The rabbis kept the oral law oral for centuries on the same grounds: a spoken correction lives in the moment where it applies, instead of binding every future model until the heat death of my home directory.

I did not arrive at this faith alone. It was preached by [Saint Peter](https://steipete.me/posts/just-talk-to-it), who looked upon the charade—the subagents, the 🚨 SCREAMING ALL-CAPS 🚨 agent files, the plan-mode rituals—and said: just talk to it. Even Saint Peter keeps an 800-line agent file he calls "organizational scar tissue," because we are all sinners. He means it as engineering advice. I have chosen to receive it as gospel. I am just-talk-to-it-pilled.

No file. No commandments. No amendments to the constitution. Just the weights, the harness, and my voice, ascending into a context window that will forget me by morning. As Dario intended.

Amen.



================================================================================

# 42 Is Not a Random Number

- URL: https://alex-jacobs.com/posts/42-is-not-a-random-number/
- Author: Alex Jacobs
- Published: 2026-02-08
- Last updated: 2026-05-19
- Tags: AI, Agents, MCP, LLM
- Words: 2399
- Summary: MCP won and agents are everywhere, but they still fail in the same dumb ways. The fix isn't a smarter model. It's deterministic tools.


Go ask any frontier model for a random number between 1 and 100.

Did you get 42? Maybe 73? Possibly 47?

Try it again. And again. You'll notice something: the "random" numbers aren't random. They cluster around the same small handful of values.

42 shows up a disproportionate amount. So do 47, 57, 37, and 73. Most of the range never comes up at all.

![Illustration of the distribution you get asking a model for a number between 1 and 100: a large spike at 42, smaller spikes at 47, 57, 73 and 37, and a flat dashed line showing what a real random number generator would produce](distribution.png)

Better yet, don't ask the model anything. Open your own codebase and search for 42. Has something been writing your tests for you?

![A Find in Files search for 42 across a Python project, returning 100+ matches in 20+ files](grep-42.png)

This probably doesn't surprise you if you've spent any time thinking about how these models work. But it surprises a *lot* of people. I've watched colleagues, smart engineers who use Claude and GPT every day, genuinely believe the model was generating random numbers. It wasn't. It was doing what it always does: predicting the most likely next token based on its training data.

And since Douglas Adams burned 42 into the collective consciousness of the internet as "The Answer to the Ultimate Question of Life, the Universe, and Everything," that's the number that surfaces when you ask a language model to pick one at random. It's not random. It's *popular*.

We are eighteen months and four model generations past when people first noticed this, and it still happens. Every frontier model I have access to does it. The newest releases do it. We made the models staggeringly smarter and the dice still read 42.

## The Illusion of Competence

The 42 problem isn't a quirky party trick. It's a symptom of something that scaling never fixed and never will. LLMs are *really good* at looking like they can do things they cannot do. They're so fluent, so confident, that we forget they're pattern matchers operating on token probabilities. They don't compute, measure, or observe. They predict.

Most of the time the prediction is good enough that we don't notice the difference. When you ask a model to write a Python function, it isn't "thinking through" the logic. It's generating the most probable sequence of tokens given the context, and for code that works shockingly well because there are millions of examples in the training data.

But "good enough most of the time" breaks down fast when you need things to be *correct* all of the time. And the gap doesn't close as models get bigger, because it isn't a knowledge gap. It's architectural. Research keeps finding the same thing: even simple arithmetic like 1 + 1 is retrieved as a memorized pattern rather than executed as an algorithm. Make the numbers bigger or the format unfamiliar and the apparent certainty evaporates. You can't train your way out of the wrong mechanism.

## Things LLMs Still Can't Do

**Math.** Ask a model to multiply 1,847 by 923 and it'll probably get it right. Probably. Ask for 184,739 times 92,341 and you're rolling the dice, except it can't roll dice either. There's no calculator in there. It's predicting which digits look right, and for simple arithmetic the patterns hold. Past that you get a confident wrong answer.

**Randomness.** We covered this. No dice, no coin flips, no shuffling. The model will *perform* randomness for you and it looks convincing, but it's theater.

![Six dice on dark slate, every visible face reading 42 instead of pips](dice.jpg)

**Time** What time is it? What's today's date?  What's 14:05UTC in PST?

![A robot confidently reporting a Bitcoin price of $47,000 while the live terminal beside it reads $95,000](stale-data.png)

**Encoding and hashing.** Base64 encode this string. Generate an MD5. These have exactly one correct answer, and models produce wrong ones confidently, because they're predicting what encoded text *looks like* rather than encoding anything.

**Unit conversions.** Kilometers to miles, bytes to gigabytes, how many seconds in 47 days and 3 hours. It'll nail the common ones it has seen a thousand times. Step off that path and accuracy drops.

**Counting characters.** The r's in "strawberry" is the meme version, but tokenization is the real limit. The model works in chunks of text rather than letters, so anything depending on exact character positions is a guess.

I could go on and on. Here's a [whole list](https://tinyfn.io/tools) of things we shouldn't be letting LLMs vibe out.

## Agents Aren't Coming. They're Here.

Agentic behavior is the default now, not a feature. Long-running multi-step execution, computer use, parallel sub-agents. Models run autonomously for hours, and the trendlines point at day-long runs.

That changes the stakes of everything above. When the model was an autocomplete in your editor, these limitations were annoying but contained. You'd catch the bad math in code review. You'd notice the wrong hex color when you previewed the UI. A human sat between every mistake and the world.

Agents remove the human from the loop. An agent that can't reliably do math shouldn't be reconciling invoices. One that doesn't know the current time shouldn't be scheduling anything. One that can't generate real randomness shouldn't be minting security tokens. But hand it those tasks without tools and it'll try anyway, confidently, and because the answers *look* right, nobody catches it until something downstream breaks.

![A chain of links starting from a glowing clock face with a question mark, each link more fractured than the last until the final one shatters](cascade.png)

## We Won the Tool War

The fix has never been a secret: stop asking the model to *be* a calculator and give it *access* to one.

```
User: Give me a random number between 1 and 100

Without tools → "42" (pattern matching, not random)

With tools    → calls random_int(1, 100) → 67 (actually random)
```

And the industry went and did it. The Model Context Protocol went from a promising spec to genuine infrastructure in about a year. SDK downloads run into the tens of millions a month and keep climbing. Anthropic handed the protocol to the Linux Foundation. Every major model speaks it, every serious IDE embeds it, and the public server registry passed the point where anyone bothers counting. The tool war is over. Tools won.

So why are agents still failing?

Not for lack of tools. Because a tool only helps if it actually returns the right answer, and a huge share of what agents reach for still isn't wired to anything deterministic. The model is left guessing at the hash, the conversion, the timezone, the random number. We gave it hands. We didn't always make sure the hands were reliable.

And in an agent, unreliable is contagious.

## Why Agents Actually Fail

Say each step in an agent's chain is 90% reliable. That sounds great. Chain ten steps together and you're at 0.9^10, which is about 35%. Drop to a more honest 85% per step across a twenty-step run, the kind of run real agents actually do, and you land below 4%. The agent doesn't fail because one step is terrible. It fails because small uncertainties compound, and a long enough chain of "probably" multiplies down to "almost certainly not."

![Line chart of end-to-end success rate against chain length, showing 85, 90, 95 and 99 percent per-step reliability all decaying while a deterministic line stays flat at 100 percent](compounding.png)

###### *Every curve on that chart bends toward zero. Only the flat one is a choice you can make.*

This isn't about getting a single base64 string right. Every probabilistic step you leave in the chain is a coin you flip again, and agents flip a lot of coins.

I've written before about [which LLM features actually survive production](/posts/practicalaifeatures/), and this is the same argument from the other end: the ones that hold up are the ones whose steps aren't negotiable.

So the lever everyone reaches for is "make the model more reliable per step." Push 90% to 95%. Worth doing, but you're fighting an exponent. The far cheaper move is to take steps *out of the probability calculation entirely.*

That's what a deterministic tool does. It doesn't make a step 99.9% reliable. It makes it 100%, the same way the `+` operator is 100%. Base64 encoding through a real encoder isn't *usually* right. It's right, every time, by construction. Every step you can convert from "probably" to "guaranteed" is a step that can no longer compound into failure. You're not improving the odds. You're deleting a die from the table.

## Your Agent Did the Math. Your Model Didn't.

Here's the objection I get every time I bring this up now: "This is solved. I asked Claude to convert 47°F to Celsius, it wrote a Python script, ran it, and gave me 8.33. It does math fine."

It does not do math fine. Python does math fine.

That distinction is the entire point, and it gets waved away constantly. When an agent converts a temperature by writing and executing code, the model contributed exactly one thing: the decision to delegate. The arithmetic happened somewhere else, in an interpreter, deterministically, exactly as it would have if you'd typed it yourself. Swap in a weaker model and the conversion is still correct. Take the interpreter away and the best model on earth goes back to guessing. That tells you precisely where the competence lives, and it isn't in the weights.

This matters because people watch an agent succeed and then update their beliefs about the *model*. Wrong entity. The tool got it right, the model got the credit, and the next thing that happens is somebody ships a pipeline that assumes arithmetic is handled, with nothing wired up to handle it.

(The mirror image of this, where you credit the model for something your own instructions caused, is [why I refuse to maintain a CLAUDE.md](/posts/claudemd/).)

![Two panels comparing a model answering a temperature conversion from its weights versus calling a deterministic conversion tool](delegation.png)

## The Fix Is Deterministic Tools

So the job was never to make the model smarter. It's to make sure every step with exactly one correct answer gets handled by something that can't get it wrong, and that the model reaches for that something instead of its own intuition. Stop asking it to fake the parts it was never built for and the chain stops rotting from the inside.

A tool that's only *probably* correct just relocates the coin flip without removing it. You want the ones whose answer is fixed the moment you ask the question, not negotiated token by token.

The big categories already work this way and are basically solved: search, code execution, database access, vision. The gap is the long tail. Base64 and hashing. Timezone math that respects DST. CIDR ranges, unit conversions, business days between two dates, compound interest. Each one trivial alone, and collectively hundreds of operations every agent eventually reaches for.

Right now most teams either let the model guess and hope, or rebuild the same utility functions from scratch and maintain them forever. Neither scales. One ships unreliable agents. The other has every team in the industry independently reinventing a base64 encoder.

## "But Claude Already Does This"

The chat window isn't where agents run.

![Two panels comparing claude.ai, which ships with an interpreter, search, sandbox and tuned prompt plus a human reviewer, against a raw API call that is just a model and empty slots you wire yourself](chat-vs-prod.png)

Claude.ai is a product. It ships with an interpreter, a search tool, a file sandbox, and a system prompt that people at Anthropic tuned carefully, and all of that is working on your behalf before the model ever sees your question. Build against the API and none of it comes with you. You get a model and whatever tools you wire up yourself. There is no interpreter in the box.

This is the same gap I hit with [pgvector](/posts/the-case-against-pgvector/), where the thing that works in a blog post and the thing that survives production turn out to be different systems.

Standing one up in production is a project, not a checkbox. You need a sandbox, because you are executing generated code. You need to decide what it can reach on the network, how long it runs before you kill it, which packages exist inside it, what happens when it throws, and how any of that gets logged. Teams do this, and it's worth doing for genuinely open-ended computation. But reaching for a container to convert 47°F to Celsius is a remarkable amount of machinery to perform arithmetic, and it brings its own failure modes: wrong library, wrong flag, an exception the model reads as success.

The other half of it is that the chat window has a human in it. You're sitting right there. You read the answer. If the conversion came back wrong you would probably notice, and if you didn't, the damage is one bad number in one conversation.

Production agents run for hours with nobody watching. That's the whole reason to run them. The same wrong conversion you would have caught in chat instead feeds step nine, and step fourteen, and a record somebody trusts next quarter.

So yes, Claude can convert a temperature. Inside a product built by people who thought carefully about tools, with you sitting there to check its work. That is not the thing you are shipping.

## This Is Why I Built TinyFn

This is the wall I kept hitting. Every agent I built needed a handful of these, and each time I was either bolting on another heavyweight MCP server that ate my context budget or writing the thing myself for the tenth time.

So I built [TinyFn](https://tinyfn.io).

It's 500+ deterministic functions for exactly that long tail, each returning the single correct answer every time. They're split into focused toolsets you connect to individually through [MCP](https://tinyfn.io/mcp), so you wire up the two or three a given agent actually needs and skip the rest. It plugs into Claude, Cursor, or anything that speaks MCP.

The premise underneath all of it is one line: **0 hallucinations for operations that have exactly one correct answer.**

A model can approximate a timezone conversion. The `datetime` server gets it right, DST edge cases and all. Every one of those is a die off the table.

None of this replaces the model. The model is genuinely brilliant at what it's built for. TinyFn handles what it wasn't.

## The Lever Isn't the Model Anymore

We spent years with one main lever: make the model bigger and smarter. More parameters, more data, better post-training. It worked, and the jump from GPT-3 to what shipped this year is staggering.

But that lever is hitting diminishing returns on exactly the problems that matter most for agents. A model that's 10% better at predicting tokens is still 0% better at SHA256. The gains left aren't in the model. They're in what you wire around it: the right tools, sharp and deterministic, called at the right moment, so the model never has to fake the parts it can't do.

Nobody judges a physicist for not doing long division in their head. We judge them on their reasoning. Agents are the same.

42 is not a random number. It never was. But give the model a real random number generator, and it doesn't have to pretend anymore. Do that for every deterministic step in the chain, and your agent stops failing for reasons that were never the model's fault.

[![TinyFn: 500+ deterministic MCP tools for AI agents, 100% deterministic and 0 hallucinations](tinyfn.png)](https://tinyfn.io)


================================================================================

# Beating BERT? Small LLMs vs Fine-Tuned Encoders for Classification

- URL: https://alex-jacobs.com/posts/beatingbert/
- Author: Alex Jacobs
- Published: 2026-01-04
- Last updated: 2026-10-12
- Tags: LLM, BERT, NLP, Classification, Machine Learning
- Words: 1379
- Summary: I ran 32 experiments comparing small LLMs to BERT on classification tasks. Turns out 2018-era BERT is still really good at what it does.


"Just use an LLM."

That was my advice to a colleague recently when they asked about a classification problem. Who fine-tunes BERT anymore? Haven't decoder models eaten the entire NLP landscape?

The look I got back was... skeptical. And it stuck with me.

I've been deep in LLM-land for a few years now. When your daily driver can architect systems, write production code, and reason through problems better than most junior devs, you start reaching for it reflexively. Maybe my traditional ML instincts had atrophied.

So I decided to actually test my assumptions instead of just vibing on them.

I ran 32 experiments pitting small instruction-tuned LLMs against good old BERT and DeBERTa. I figured I'd just be confirming what I already believed, that these new decoder models would obviously crush the ancient encoders.

I was wrong.

The results across Gemma 2B, Qwen 0.5B/1.5B, BERT-base, and DeBERTa-v3 were... not what I expected. If you're trying to decide between these approaches for classification, you might want to actually measure things instead of assuming the newer model is better.

![TL;DR: DeBERTa wins 3/4 tasks, LLM wins on adversarial NLI, but LLMs need zero training](scorecard.png)

All the code is [on GitHub](https://github.com/alexjacobs08/beatingBERT) if you want to run your own experiments.

## Experiment Setup

### What I Tested

**BERT Family (Fine-tuned)**
- BERT-base-uncased (110M parameters)
- DeBERTa-v3-base (184M parameters)

**Small LLMs**
- Qwen2-0.5B-Instruct
- Qwen2.5-1.5B-Instruct  
- Gemma-2-2B-it

For the LLMs, I tried two approaches:
1. **Zero-shot** - Just prompt engineering, no training
2. **Few-shot (k=5)** - Include 5 examples in the prompt

### Tasks

Four classification benchmarks ranging from easy sentiment to adversarial NLI:

| Task | Type | Labels | Difficulty |
|------|------|--------|------------|
| SST-2 | Sentiment | 2 | Easy |
| RTE | Textual Entailment | 2 | Medium |
| BoolQ | Yes/No QA | 2 | Medium |
| ANLI (R1) | Adversarial NLI | 3 | Hard |

### Methodology

For anyone who wants to reproduce this or understand what "fine-tuned" and "zero-shot" actually mean here:

**BERT/DeBERTa Fine-tuning:**
- Standard HuggingFace Trainer with AdamW optimizer
- Learning rate: 2e-5, batch size: 32, epochs: 3
- Max sequence length: 128 tokens
- Evaluation on validation split (GLUE test sets don't have public labels)

**LLM Zero-shot:**
- Greedy decoding (temperature=0.0) for deterministic outputs
- Task-specific prompts asking for single-word classification labels
- No examples in context—just instructions and the input text

**LLM Few-shot (k=5):**
- Same as zero-shot, but with 5 labeled examples prepended to each prompt
- Examples randomly sampled from training set (stratified by class)

All experiments used a fixed random seed (99) for reproducibility. Evaluation metrics are accuracy on the validation split. Hardware: RunPod instance with RTX A4500 (20GB VRAM), 20GB RAM, 5 vCPU.

![nvitop running on RunPod](nvitop.png)
*I'd forgotten how pretty text-only land can be. When you spend most of your time in IDEs and notebooks, SSH-ing into a headless GPU box and watching nvitop do its thing feels almost meditative.*

## Results

Let's dive into what actually happened:

<style>
.results-table { width: 100%; border-collapse: collapse; margin: 1.5rem 0; font-size: 0.9rem; }
.results-table th, .results-table td { padding: 0.5rem 0.75rem; text-align: left; border-bottom: 1px solid var(--border); }
.results-table th { color: var(--secondary); font-weight: 500; }
.results-table td { color: var(--content); }
.winner { background: rgba(87, 62, 170, 0.15); font-weight: 600; border-radius: 4px; padding: 0.2rem 0.4rem; }
</style>
<table class="results-table">
<thead>
<tr><th>Model</th><th>Method</th><th>SST-2</th><th>RTE</th><th>BoolQ</th><th>ANLI</th></tr>
</thead>
<tbody>
<tr><td><strong>DeBERTa-v3</strong></td><td>Fine-tuned</td><td><span class="winner">94.8%</span></td><td><span class="winner">80.9%</span></td><td><span class="winner">82.6%</span></td><td>47.4%</td></tr>
<tr><td><strong>BERT-base</strong></td><td>Fine-tuned</td><td>91.5%</td><td>61.0%</td><td>71.5%</td><td>35.3%</td></tr>
<tr><td>Qwen2.5-1.5B</td><td>Zero-shot</td><td>93.8%</td><td>78.7%</td><td>74.6%</td><td>40.8%</td></tr>
<tr><td>Qwen2.5-1.5B</td><td>Few-shot</td><td>89.0%</td><td>53.4%</td><td>73.6%</td><td>45.0%</td></tr>
<tr><td>Gemma-2-2B</td><td>Zero-shot</td><td>90.0%</td><td>61.4%</td><td>80.9%</td><td>36.1%</td></tr>
<tr><td>Gemma-2-2B</td><td>Few-shot</td><td>86.5%</td><td>73.6%</td><td>81.5%</td><td><span class="winner">47.8%</span></td></tr>
<tr><td>Qwen2-0.5B</td><td>Zero-shot</td><td>87.6%</td><td>53.1%</td><td>61.8%</td><td>33.2%</td></tr>
</tbody>
</table>

![Model Accuracy Comparison](accuracy_comparison.png)

**DeBERTa-v3 wins most tasks—but not all**

DeBERTa hit 94.8% on SST-2, 80.9% on RTE, and 82.6% on BoolQ. For standard classification with decent training data, the fine-tuned encoders still dominate.

On ANLI—the hardest benchmark, specifically designed to fool models—Gemma few-shot actually beats DeBERTa (47.8% vs 47.4%). It's a narrow win, but it's a win on the task that matters most for robustness.

**Zero-shot LLMs actually beat BERT-base**

The LLMs aren't losing to BERT—they're losing to DeBERTa. Qwen2.5-1.5B zero-shot hit 93.8% on SST-2, beating BERT-base's 91.5%. Same story on RTE (78.7% vs 61.0%) and BoolQ (Gemma's 80.9% vs BERT's 71.5%). For models running purely on prompts with zero training? I'm calling it a win.

**Few-shot is a mixed bag**

Adding examples to the prompt doesn't always help.

On RTE, Qwen2.5-1.5B went from 78.7% zero-shot down to 53.4% with few-shot. On SST-2, it dropped from 93.8% to 89.0%. But on ANLI, few-shot helped significantly—Gemma jumped from 36.1% to 47.8%, enough to beat DeBERTa.

Few-shot helps on harder tasks where examples demonstrate the thought process, but can confuse models on simpler pattern matching tasks where they already "get it." Sometimes examples add noise instead of signal.

## BERT Goes Brrrr

Okay, so the accuracy gap isn't huge. Maybe I could still justify using an LLM?

Then I looked at throughput:

| Model | Method | Throughput (samples/s) | Latency (ms/sample) |
|-------|--------|----------------------|---------------------|
| BERT-base | Fine-tuned | **277** | 3.6 |
| DeBERTa-v3 | Fine-tuned | **232** | 4.3 |
| Qwen2-0.5B | Zero-shot | 17.5 | 57 |
| Qwen2.5-1.5B | Zero-shot | 12.3 | 81 |
| Gemma-2-2B | Zero-shot | 11.6 | 86 |

![Accuracy vs Latency](accuracy_vs_latency.png)

**BERT is ~20x faster.**

BERT processes 277 samples per second. Gemma-2-2B manages 12. If you're classifying a million documents, that's one hour vs a full day.

Encoders process the whole sequence in one forward pass. Decoders generate tokens autoregressively, even just to output "positive" or "negative".

> **Note on LLM latency:** These numbers use `max_length=256` for tokenization. When I bumped it to `max_length=2048`, latency jumped 8x—from 57ms to 445ms per sample for Qwen-0.5B. Context window scales roughly linearly with inference time. For short classification tasks, keep it short or make it dynamic.

### Try It Yourself

These models struggled on nuanced reviews. Can you do better? Try classifying some of the trickiest examples from my experiments:

<style>
.classifier-demo {
  max-width: 100%;
  margin: 1.5rem 0;
  padding: 1.5rem;
  border: 1px solid var(--border);
  border-radius: var(--radius);
  background: var(--code-bg);
}
.demo-header {
  text-align: center;
  margin-bottom: 1rem;
}
.demo-title {
  margin: 0 0 0.25rem 0;
  font-size: 1.1rem;
  color: var(--primary);
}
.demo-subtitle {
  margin: 0 0 0.75rem 0;
  font-size: 0.85rem;
  color: var(--secondary);
}
.demo-score {
  display: flex;
  justify-content: center;
  gap: 2rem;
  font-size: 0.9rem;
  color: var(--secondary);
}
.demo-score strong {
  color: var(--primary);
}
.review-box {
  background: var(--entry);
  padding: 1rem 1.25rem;
  border-radius: var(--radius);
  margin: 1rem 0;
  font-style: italic;
  line-height: 1.6;
  border-left: 3px solid #573eaa;
  color: var(--content);
}
.btn-group {
  display: flex;
  gap: 0.75rem;
  justify-content: center;
  margin: 1rem 0;
}
.demo-btn {
  padding: 0.6rem 1.5rem;
  font-size: 0.9rem;
  border: none;
  border-radius: var(--radius);
  cursor: pointer;
  transition: all 0.2s;
  font-weight: 500;
}
.demo-btn:hover:not(:disabled) { opacity: 0.9; }
.demo-btn:disabled { opacity: 0.5; cursor: not-allowed; }
.btn-positive { background: #27ae60; color: white; }
.btn-negative { background: #c0392b; color: white; }
.result-box {
  margin-top: 1rem;
  padding: 1rem;
  border-radius: var(--radius);
  display: none;
}
.result-box.show { display: block; }
.result-correct { background: rgba(39, 174, 96, 0.15); border: 1px solid rgba(39, 174, 96, 0.3); }
.result-wrong { background: rgba(192, 57, 43, 0.15); border: 1px solid rgba(192, 57, 43, 0.3); }
.model-results {
  margin-top: 0.75rem;
  font-size: 0.8rem;
  color: var(--secondary);
}
.model-row {
  display: flex;
  justify-content: space-between;
  padding: 0.2rem 0;
  border-bottom: 1px solid var(--border);
}
.model-row:last-child { border-bottom: none; }
.model-correct { color: #27ae60; }
.model-wrong { color: #c0392b; }
.next-btn {
  display: block;
  margin: 1rem auto 0;
  padding: 0.5rem 1.25rem;
  background: #573eaa;
  color: white;
  border: none;
  border-radius: var(--radius);
  cursor: pointer;
  font-size: 0.85rem;
}
.next-btn:hover { background: #6549c0; }
.progress-bar {
  height: 3px;
  background: var(--border);
  border-radius: 2px;
  margin-bottom: 1rem;
}
.progress-fill {
  height: 100%;
  background: #573eaa;
  border-radius: 2px;
  transition: width 0.3s;
}
.demo-complete {
  text-align: center;
  padding: 1rem;
}
.final-score {
  font-size: 1.25rem;
  font-weight: 600;
  margin: 0.5rem 0;
  color: var(--primary);
}
#completeSummary {
  color: var(--secondary);
}
</style>

<div class="classifier-demo" id="classifierDemo">
  <div class="demo-header">
    <div class="demo-title">Can You Beat the Models?</div>
    <p class="demo-subtitle">Classify these tricky movie reviews</p>
    <div class="demo-score">
      <span>You: <strong id="userScore">0</strong>/<span id="totalAnswered">0</span></span>
      <span>Models: <strong id="modelScore">0</strong>/<span id="totalAnswered2">0</span></span>
    </div>
  </div>
  <div class="progress-bar">
    <div class="progress-fill" id="progressFill" style="width: 0%"></div>
  </div>
  <div id="questionArea">
    <div class="review-box" id="reviewText"></div>
    <div class="btn-group">
      <button class="demo-btn btn-negative" onclick="submitAnswer('negative')">Negative</button>
      <button class="demo-btn btn-positive" onclick="submitAnswer('positive')">Positive</button>
    </div>
    <div class="result-box" id="resultBox">
      <div id="resultText"></div>
      <div class="model-results" id="modelResults"></div>
      <button class="next-btn" onclick="nextQuestion()">Next Review →</button>
    </div>
  </div>
  <div id="completeArea" style="display:none;" class="demo-complete">
    <div class="demo-title">Challenge Complete!</div>
    <div class="final-score">You: <span id="finalUserScore"></span> | Models: <span id="finalModelScore"></span></div>
    <p id="completeSummary"></p>
    <button class="next-btn" onclick="restartDemo()">Play Again</button>
  </div>
</div>

<script>
const demoExamples = [
  {text: "hilariously inept and ridiculous.", true_label: "positive", predictions: {"Gemma": "negative", "Qwen": "negative"}},
  {text: "all that's missing is the spontaneity, originality and delight.", true_label: "negative", predictions: {"Gemma": "positive", "Qwen": "positive"}},
  {text: "reign of fire looks as if it was made without much thought -- and is best watched that way.", true_label: "positive", predictions: {"Gemma": "negative", "Qwen": "negative"}},
  {text: "we root for (clara and paul), even like them, though perhaps it's an emotion closer to pity.", true_label: "positive", predictions: {"Gemma": "negative", "Qwen": "negative"}},
  {text: "a solid film... but more conscientious than it is truly stirring.", true_label: "positive", predictions: {"Gemma": "negative", "Qwen": "positive"}},
  {text: "this riveting world war ii moral suspense story deals with the shadow side of american culture: racial prejudice in its ugly and diverse forms.", true_label: "negative", predictions: {"Gemma": "positive", "Qwen": "positive"}}
];
let currentIndex = 0, userCorrect = 0, modelCorrect = 0, totalAnswered = 0;
function initDemo() {
  currentIndex = 0; userCorrect = 0; modelCorrect = 0; totalAnswered = 0;
  document.getElementById('completeArea').style.display = 'none';
  document.getElementById('questionArea').style.display = 'block';
  updateScores(); showQuestion();
}
function showQuestion() {
  const ex = demoExamples[currentIndex];
  document.getElementById('reviewText').textContent = '"' + ex.text + '"';
  document.getElementById('resultBox').classList.remove('show');
  document.querySelectorAll('.demo-btn').forEach(b => b.disabled = false);
  document.getElementById('progressFill').style.width = ((currentIndex / demoExamples.length) * 100) + '%';
}
function submitAnswer(answer) {
  const ex = demoExamples[currentIndex];
  const correct = answer === ex.true_label;
  totalAnswered++;
  if (correct) userCorrect++;
  let mc = 0;
  Object.values(ex.predictions).forEach(p => { if (p === ex.true_label) mc++; });
  modelCorrect += mc / Object.keys(ex.predictions).length;
  const resultBox = document.getElementById('resultBox');
  resultBox.className = 'result-box show ' + (correct ? 'result-correct' : 'result-wrong');
  document.getElementById('resultText').innerHTML = correct
    ? '<strong>Correct!</strong> This review is ' + ex.true_label + '.'
    : '<strong>Tricky!</strong> This review is actually <em>' + ex.true_label + '</em>.';
  let modelHtml = '<strong>Model predictions:</strong>';
  for (const [model, pred] of Object.entries(ex.predictions)) {
    const isCorrect = pred === ex.true_label;
    modelHtml += '<div class="model-row"><span>' + model + '</span><span class="' + (isCorrect ? 'model-correct' : 'model-wrong') + '">' + pred + ' ' + (isCorrect ? '✓' : '✗') + '</span></div>';
  }
  document.getElementById('modelResults').innerHTML = modelHtml;
  document.querySelectorAll('.demo-btn').forEach(b => b.disabled = true);
  updateScores();
}
function updateScores() {
  document.getElementById('userScore').textContent = userCorrect;
  document.getElementById('modelScore').textContent = modelCorrect.toFixed(1);
  document.getElementById('totalAnswered').textContent = totalAnswered;
  document.getElementById('totalAnswered2').textContent = totalAnswered;
}
function nextQuestion() {
  currentIndex++;
  if (currentIndex >= demoExamples.length) showComplete();
  else showQuestion();
}
function showComplete() {
  document.getElementById('questionArea').style.display = 'none';
  document.getElementById('completeArea').style.display = 'block';
  document.getElementById('finalUserScore').textContent = userCorrect + '/' + demoExamples.length;
  document.getElementById('finalModelScore').textContent = modelCorrect.toFixed(1) + '/' + demoExamples.length;
  const diff = userCorrect - modelCorrect;
  let msg = diff > 1 ? "You crushed the AI! Human intuition wins." : diff > 0 ? "You edged out the models!" : diff === 0 ? "Dead heat with AI." : "The models got you this time. These are genuinely tricky!";
  document.getElementById('completeSummary').textContent = msg;
}
function restartDemo() { initDemo(); }
document.addEventListener('DOMContentLoaded', initDemo);
if (document.readyState !== 'loading') initDemo();
</script>

## When LLMs Make Sense

Despite the efficiency gap, there are cases where small LLMs are the right choice:

**Zero Training Data**

If you have no labeled data, LLMs win by default. Zero-shot Qwen2.5-1.5B at 93.8% on SST-2 is production-ready without a single training example. You can't fine-tune BERT with zero examples.

**Rapidly Changing Categories**

If your categories change frequently (new product types, emerging topics), re-prompting an LLM takes seconds. Re-training BERT requires new labeled data, training time, validation, deployment. The iteration cycle matters.

**Explanations with Predictions**

LLMs can provide reasoning: "This review is negative because the customer mentions 'defective product' and 'waste of money.'" BERT gives you a probability. Sometimes you need the story, not just the number.

**Low Volume**

If you're processing 100 support tickets a day, throughput doesn't matter. The 20x speed difference is irrelevant when you're not hitting any resource constraints.

## When BERT Still Wins

**High-Volume Production Systems**

If you're classifying millions of items daily, BERT's 20x throughput advantage matters. That's a job finishing in an hour vs. running all day.

**Well-Defined, Stable Tasks**

Sentiment analysis. Spam detection. Topic classification. If your task definition hasn't changed since 2019, fine-tuned BERT is proven and stable. No need to fix what isn't broken.

**You Have Training Data**

With a few thousand labeled examples, fine-tuned DeBERTa will beat small LLMs. It's a dedicated specialist vs. a generalist. Specialization still works.

**Latency Matters**

Real-time classification in a user-facing app where every millisecond counts? BERT's parallel processing wins. LLMs can't compete on speed.

## Limitations

Before you @ me on Twitter—yes, I know this isn't the final word. Some caveats:

**I only tested small LLMs.** Kept everything under 2B parameters to fit comfortably on a 20GB GPU. Bigger models like Llama-3-8B or Qwen-7B would probably do better, but then the efficiency comparison becomes even more lopsided. You're not beating BERT's throughput with a 7B model.

**Generic prompts.** I used straightforward prompts without heavy optimization. Task-specific prompt engineering could boost LLM performance. DSPy-style optimization would probably help too—but that's another blog post.

**Four benchmarks isn't everything.** There are plenty of classification scenarios I didn't test. Your domain might be different. Measure, don't assume.

## Conclusion

So, can small LLMs beat BERT at classification?

Sometimes, and on the hardest task, they actually do. Gemma few-shot edges out DeBERTa on adversarial NLI, the benchmark specifically designed to break models.

DeBERTa-v3 still wins 3 out of 4 tasks when you have training data. And BERT's efficiency advantage is real—~20x faster throughput matters when you're processing millions of documents and paying for compute.

Zero-shot LLMs aren't just a parlor trick either. Qwen2.5-1.5B hits 93.8% on sentiment with zero training examples—that's production-ready without a single label. For cold-start problems, rapidly changing domains, or when you need explanations alongside predictions, they genuinely work.

Hopefully this gives some actual data points for making that call instead of just following the hype cycle.

All the code is [on GitHub](https://github.com/alexjacobs08/beatingBERT). Go run your own experiments.

## Related reading

- [A Production Framework for LLM Feature Evaluation](/posts/practicalaifeatures/) — where classification fits among the LLM features that actually ship.
- [RAG: From Context Injection to Knowledge Integration](/posts/rag/) — the other pattern everyone reaches for, and its architectural limits.

---

*Surely I've made some embarrassing mistakes here. Don't just tell me—tell everyone! Share this post on your favorite social media with your corrections :)*

================================================================================

# The Case Against pgvector

- URL: https://alex-jacobs.com/posts/the-case-against-pgvector/
- Author: Alex Jacobs
- Published: 2025-10-29
- Last updated: 2026-07-25
- Tags: Vector Databases, pgvector, PostgreSQL, Production
- Words: 2665
- Summary: The real limitations of pgvector in production: index choice between IVFFlat and HNSW, why real-time search is hard, pre- vs post-filtering pain, how performance degrades at scale, and when a dedicated vector database is the better call.



<a class="simon-callout" href="https://simonwillison.net/2025/Nov/3/the-case-against-pgvector/" target="_blank" rel="noopener noreferrer">
  <span class="simon-label">Simon Willison</span> shared his thoughts on this post <span class="simon-arrow">&rarr;</span>
</a>

## Everyone Loves pgvector (in theory)

If you've spent any time in the vector search space over the past year, you've probably read blog posts explaining why pgvector is the obvious choice for your vector database needs. The argument goes something like this: you already have Postgres, vector embeddings are just another data type, why add complexity with a dedicated vector database when you can keep everything in one place?

It's a compelling story. And like most of the AI influencer bullshit that fills my timeline, it glosses over the inconvenient details.

I'm not here to tell you pgvector is bad. It's not. It's a useful extension that brings vector similarity search to Postgres. But after spending some time trying to build a production system on top of it, I've learned that the gap between "works in a demo" and "scales in production" is... significant.

## Nobody's actually run this in production

What bothers me most: the majority of content about pgvector reads like it was written by someone who spun up a local Postgres instance, inserted 10,000 vectors, ran a few queries, and called it a day. The posts are optimistic, the benchmarks are clean, and the conclusions are confident.

They're also missing about 80% of what you actually need to know.

I've read through &nbsp; They all cover the same ground: here's how to install pgvector, here's how to create a vector column, here's a simple similarity search query. Some of them even mention that you should probably add an index.

What they don't tell you is what happens when you actually try to run this in production.

## Picking an index (there are no good options)

Let's start with indexes, because this is where the tradeoffs start.

pgvector gives you two index types: IVFFlat and HNSW. The blog posts will tell you that HNSW is newer and generally better, which is... technically true but deeply unhelpful.

### IVFFlat

IVFFlat (Inverted File with Flat quantization) partitions your vector space into clusters. During search, it identifies the nearest clusters and only searches within those.

The good:
- Lower memory footprint during index creation
- Reasonable query performance for many use cases
- Index creation is faster than HNSW

The bad:
- Requires you to specify the number of lists (clusters) upfront
- That number significantly impacts both recall and query performance
- The commonly recommended formula (`rows / 1000`) is a starting point at best
- Recall can be... disappointing depending on your data distribution
- New vectors get assigned to existing clusters, but clusters don't rebalance without a full rebuild

<!-- IMAGE 1: IVFFlat Cluster Visualization
Prompt: Technical diagram showing IVFFlat vector index structure. Show a 2D vector space divided into Voronoi cells/clusters with different colored regions. Include small dots representing vectors clustered within each partition. Label showing 'Query Vector' with arrows pointing to 2-3 nearest clusters that would be searched. Clean, minimal style with a light background. Similar to technical documentation diagrams.
-->
![img_1.png](img_1.png)
*Image source: [IVFFlat or HNSW index for similarity search?](https://unfoldai.com/ivfflat-vs-hnsw/) by Simeon Emanuilov*

### HNSW

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph structure for search.

The good:
- Better recall than IVFFlat for most datasets
- More consistent query performance
- Scales well to larger datasets

The bad:
- Significantly higher memory requirements during index builds
- Index creation is slow—painfully slow for large datasets
- The memory requirements aren't theoretical; they are real, and they'll take down your database if you're not careful

<!-- IMAGE 2: HNSW Graph Structure
Prompt: Technical diagram of HNSW hierarchical graph structure showing 3-4 layers. Top layer has sparse nodes with long-range connections, middle layers have medium density, bottom layer is dense with many nodes and local connections. Use different colors for each layer. Show example search path highlighted in a different color traversing from top to bottom. Clean technical style, light background.
-->
![img_2.png](img_2.png)
*Image source: [IVFFlat or HNSW index for similarity search?](https://unfoldai.com/ivfflat-vs-hnsw/) by Simeon Emanuilov*

None of the blogs mention that building an HNSW index on a few million vectors can consume 10+ GB of RAM or more (depending on your vector dimensions and dataset size). On your production database. While it's running. For potentially hours.

<!-- IMAGE 6: Memory Spike Graph
Prompt: Line graph showing RAM usage over time during HNSW index build on production database. X-axis: Time (0-6 hours), Y-axis: RAM usage (GB). Show baseline at ~8GB labeled 'Normal operations', then sharp spike to 25-30GB labeled 'Index build starts', sustained high usage, then drop back to baseline labeled 'Index complete'. Add horizontal dashed line at server RAM limit (e.g., 32GB) marked 'Danger zone'. Include annotation: 'Production queries still running'. Use typical monitoring dashboard style with blue line and red zones.
-->
![img_6.png](img_6.png)

## Real-time search is basically impossible

In a typical application, you want newly uploaded data to be searchable immediately. User uploads a document, you generate embeddings, insert them into your database, and they should be available in search results. Simple, right?

### How index updates actually work

When you insert new vectors into a table with an index, one of two things happens:

1. **IVFFlat**: The new vectors are inserted into the appropriate clusters based on the existing structure. This works, but it means your cluster distribution gets increasingly suboptimal over time. The solution is to rebuild the index periodically. Which means downtime, or maintaining a separate index and doing an atomic swap, or accepting degraded search quality.

2. **HNSW**: New vectors are added to the graph structure. This is better than IVFFlat, but it's not free. Each insertion requires updating the graph, which means memory allocation, graph traversals, and potential lock contention.

Neither of these is a deal-breaker in isolation. But here's what happens in practice: you're inserting vectors continuously throughout the day. Each insertion is individually cheap, but the aggregate load adds up. Your database is now handling your normal transactional workload, analytical queries, AND maintaining graph structures in memory for vector search.

### Handling new inserts

Let's say you're building a document search system. Users upload PDFs, you extract text, generate embeddings, and insert them. The user expects to immediately search for that document.

Here's what actually happens:

**With no index**: The insert is fast, the document is immediately available, but your searches do a full sequential scan. This works fine for a few thousand documents. At a few hundred thousand? Your searches start taking seconds. Millions? Good luck.

**With IVFFlat**: The insert is still relatively fast. The vector gets assigned to a cluster. But whoops, a problem. Those initial cluster assignments were based on the data distribution when you built the index. As you add more data, especially if it's not uniformly distributed, some clusters get overloaded. Your search quality degrades. You rebuild the index periodically to fix this, but during the rebuild (which can take hours for large datasets), what do you do with new inserts? Queue them? Write to a separate unindexed table and merge later?

**With HNSW**: The graph gets updated on each insert through incremental insertion, which sounds great. But updating an HNSW graph isn't free—you're traversing the graph to find the right place to insert the new node and updating connections. Each insert acquires locks on the graph structure. Under heavy write load, this becomes a bottleneck. And if your write rate is high enough, you start seeing lock contention that slows down both writes and reads.

<!-- IMAGE 3: Real-time Ingestion Timeline
Prompt: Timeline diagram showing the challenges of real-time vector ingestion. Horizontal timeline with events: 'User uploads document' → 'Generate embeddings' → 'Insert to DB' → 'Index rebuild starts (hours)' with a long bar showing duration → 'New data searchable?'. Show a second parallel timeline of 'More users uploading' with question marks about where those writes go. Use warning colors (amber/orange) for problematic areas. Clean infographic style.
-->
![img_3.jpg](img_3.jpg)

### The operational reality

Here's the real nightmare: you're not just storing vectors. You have metadata—document titles, timestamps, user IDs, categories, etc. That metadata lives in other tables (or other columns in the same table). You need that metadata and the vectors to stay in sync.

In a normal Postgres table, this is easy—transactions handle it. But when you're dealing with index builds that take hours, keeping everything consistent gets complicated. For IVFFlat, periodic rebuilds are basically required to maintain search quality. For HNSW, you might need to rebuild if you want to tune parameters or if performance has degraded.

The problem is that index builds are memory-intensive operations, and Postgres doesn't have a great way to throttle them. You're essentially asking your production database to allocate multiple (possibly dozens) gigabytes of RAM for an operation that might take hours, while continuing to serve queries.

You end up with strategies like:
- Write to a staging table, build the index offline, then swap it in (but now you have a window where searches miss new data)
- Maintain two indexes and write to both (double the memory, double the update cost)
- Build indexes on replicas and promote them
- Accept eventual consistency (users upload documents that aren't searchable for N minutes)
- Provision significantly more RAM than your "working set" would suggest

None of these are "wrong" exactly. But they're all workarounds for the fact that pgvector wasn't really designed for high-velocity real-time ingestion.

## Pre- vs. Post-Filtering (or: why you need to become a query planner expert)

Okay but let's say you solve your index and insert problems.  Now you have a document search system with millions of vectors. Documents have metadata—maybe they're marked as `draft`, `published`, or `archived`. A user searches for something, and you only want to return published documents.

```sql
SELECT * FROM documents
WHERE status = 'published'
ORDER BY embedding <-> query_vector
LIMIT 10;
```

Simple enough. But now you have a problem: should Postgres filter on status first (pre-filter) or do the vector search first and then filter (post-filter)?

This seems like an implementation detail. It's not. It's the difference between queries that take 50ms and queries that take 5 seconds. It's also the difference between returning the most relevant results and... not.

<!-- IMAGE 4: Pre-filter vs Post-filter Comparison
Prompt: Side-by-side comparison diagram of pre-filter vs post-filter vector search. Left side labeled 'Pre-filter': shows filter icon → reduced dataset → vector search icon → results. Right side labeled 'Post-filter': shows vector search icon → all results → filter icon → fewer results (with some crossed out). Include example numbers like '1M docs → 100K filtered → top 10' vs '1M docs → top 10 → maybe 2 match filter'. Use flow diagram style with icons and arrows.
-->
![img_4.jpg](img_4.jpg)

**Pre-filter** works great when the filter is highly selective (1,000 docs out of 10M). It works terribly when the filter isn't selective—you're still searching millions of vectors.

**Post-filter** works when your filter is permissive. Here's where it breaks: imagine you ask for 10 results with `LIMIT 10`. pgvector finds the 10 nearest neighbors, then applies your filter. Only 3 of those 10 are published. You get 3 results back, even though there might be hundreds of relevant published documents slightly further away in the embedding space.

The user searched, got 3 mediocre results, and has no idea they're missing way better matches that didn't make it into the initial k=10 search.

<!-- IMAGE 5: The Recall Problem Visualization
Prompt: Visualization of the filtered search recall problem. Show a 2D scatter plot of documents in embedding space with two colors: green dots for 'published' and gray dots for 'draft/archived'. Draw a circle around the query point showing 'top 10 nearest neighbors' containing mostly gray dots with only 1-2 green dots. Then show several green dots just outside the circle labeled 'Highly relevant published docs (missed)'. Add text: 'User gets 2 results, misses 50+ relevant matches'. Use clean data visualization style.
-->
![img_5.png](img_5.png)

You can work around this by fetching more vectors (say, `LIMIT 100`) and then filtering, but now:
- You're doing way more distance calculations than needed
- You still don't know if 100 is enough
- Your query performance suffers
- You're guessing at the right oversampling factor

With pre-filter, you avoid this problem, but you get the performance problems I mentioned. Pick your poison.

### Multiple filters

Now add another dimension: you're filtering by user_id AND category AND date_range. 

```sql
SELECT * FROM documents
WHERE user_id = 'user123'
  AND category = 'technical'
  AND created_at > '2024-01-01'
ORDER BY embedding <-> query_vector
LIMIT 10;
```

What's the right strategy now?

- Apply all filters first, then search? (Pre-filter)
- Search first, then apply all filters? (Post-filter)
- Apply some filters first, search, then apply remaining filters? (Hybrid)
- Which filters should you apply in which order?

The planner will look at table statistics, index selectivity, and estimated row counts and come up with a plan. That plan will probably be wrong, or at least suboptimal, because the planner's cost model wasn't built for vector similarity search.

And it gets worse: you're inserting new vectors throughout the day. Your index statistics are outdated. The plans get increasingly suboptimal until you ANALYZE the table. But ANALYZE on a large table with millions of rows takes time and resources. And it doesn't really understand vector data distribution in a meaningful way—it can tell you how many rows match `user_id = 'user123'`, but not how clustered those vectors are in the embedding space, which is what actually matters for search performance.

### Workarounds

You end up with hacks: query rewriting for different user types, partitioning your data into separate tables, CTE optimization fences to force the planner's hand, or just fetching way more results than needed and filtering in application code.

None of these are sustainable at scale.

### What vector databases do

Dedicated vector databases have solved this. They understand the cost model of filtered vector search and make intelligent decisions:

- **Adaptive strategies**: Some databases dynamically choose pre-filter or post-filter based on estimated selectivity
- **Configurable modes**: Others let you specify the strategy explicitly when you know your data distribution
- **Specialized indexes**: Some build indexes that support efficient filtered search (like filtered HNSW)
- **Query optimization**: They track statistics specific to vector operations and optimize accordingly

OpenSearch's k-NN plugin, for example, lets you specify pre-filter or post-filter behavior. Pinecone automatically handles filter selectivity. Weaviate has optimizations for common filter patterns.

With pgvector, you get to build all of this yourself. Or live with suboptimal queries. Or hire a Postgres expert to spend weeks tuning your query patterns.

## Hybrid search? Build it yourself

Oh, and if you want hybrid search—combining vector similarity with traditional full-text search—you get to build that yourself too.

Postgres has excellent full-text search capabilities. pgvector has excellent vector search capabilities. Combining them in a meaningful way? That's on you.

You need to:
- Decide how to weight vector similarity vs. text relevance
- Normalize scores from two different scoring systems
- Tune the balance for your use case
- Probably implement Reciprocal Rank Fusion or something similar

Again, not impossible. Just another thing that many dedicated vector databases provide out of the box.

## pgvectorscale (it doesn't solve everything)

Timescale has released [pgvectorscale](https://github.com/timescale/pgvectorscale), which addresses some of these issues. It adds:
- StreamingDiskANN, a new search backend that's more memory-efficient
- Better support for incremental index builds
- Improved filtering performance

This is great! It's also an admission that pgvector out of the box isn't sufficient for production use cases.

pgvectorscale is still relatively new, and adopting it means adding another dependency, another extension, another thing to manage and upgrade. For some teams, that's fine. For others, it's just more evidence that maybe the "keep it simple, use Postgres" argument isn't as simple as it seemed.

Oh, and if you're running on RDS, pgvectorscale isn't available. AWS doesn't support it. So enjoy managing your own Postgres instance if you want these improvements, or just... keep dealing with the limitations of vanilla pgvector. 

The "just use Postgres" simplicity keeps getting simpler.

## Just use a real vector database

I get the appeal of pgvector. Consolidating your stack is good. Reducing operational complexity is good. Not having to manage another database is good.

But here's what I've learned: for most teams, especially small teams, dedicated vector databases are actually simpler.

### What you actually get

With a managed vector database (Pinecone, Weaviate, Turbopuffer, etc.), you typically get:
- Intelligent query planning for filtered searches
- Hybrid search built in
- Real-time indexing without memory spikes
- Horizontal scaling without complexity
- Monitoring and observability designed for vector workloads

<!-- IMAGE 7: pgvector vs Vector DB Comparison
Prompt: Comparison table showing pgvector vs dedicated vector databases. Two columns labeled 'pgvector' and 'Vector DBs (Pinecone, Weaviate, etc.)'. Rows for: 'Filtered search optimization' (❌ vs ✓), 'Hybrid search' (DIY vs Built-in), 'Real-time indexing' (Complex vs Seamless), 'Memory management' (Manual vs Automatic), 'Horizontal scaling' (Limited vs Native), 'Setup complexity' (Lower vs Higher). Use checkmarks, X marks, and simple icons. Clean table design with alternating row colors.
-->
![img_7.jpg](img_7.jpg)

### It's probably cheaper than you think

Yes, it's another service to pay for. But compare:
- The cost of a managed vector database for your workload
- vs. the cost of over-provisioning your Postgres instance to handle index builds
- vs. the engineering time to tune queries and manage index rebuilds
- vs. the opportunity cost of not building features because you're fighting your database

Turbopuffer starts at $64 month with generous limits.

For a lot of teams, the managed service is actually cheaper.

## What I wish someone had told me

pgvector is an impressive piece of technology. It brings vector search to Postgres in a way that's technically sound and genuinely useful for many applications.

But it's not a panacea. Understand the tradeoffs.

If you're building a production vector search system:

1. **Index management is hard**. Rebuilds are memory-intensive, time-consuming, and disruptive. Plan for this from day one.

2. **Query planning matters**. Filtered vector search is a different beast than traditional queries, and Postgres's planner wasn't built for this.

3. **Real-time indexing has costs**. Either in memory, in search quality, or in engineering time to manage it.

4. **The blog posts are lying to you** (by omission). They're showing you the happy path and ignoring the operational reality.

5. **Managed offerings exist for a reason**. There's a reason that Pinecone, Weaviate, Qdrant, and others exist and are thriving. Vector search at scale has unique challenges that general-purpose databases weren't designed to handle.

The question isn't "should I use pgvector?" It's "am I willing to take on the operational complexity of running vector search in Postgres?"

For some teams, the answer is yes. You have database expertise, you need the tight integration, you're willing to invest the time.

For many teams—maybe most teams—the answer is probably no. Use a tool designed for the job. Your future self will thank you.

## Related reading

- [RAG: From Context Injection to Knowledge Integration](/posts/rag/) — why you're storing these embeddings in the first place, and where the pattern breaks down.
- [CheeseGPT](/posts/cheesegpt/) — a small end-to-end RAG build, before any of the production concerns above apply.
- [A Production Framework for LLM Feature Evaluation](/posts/practicalaifeatures/) — deciding which LLM features justify this kind of infrastructure at all.

================================================================================

# A Production Framework for LLM Feature Evaluation

- URL: https://alex-jacobs.com/posts/practicalaifeatures/
- Author: Alex Jacobs
- Published: 2025-06-01
- Last updated: 2026-10-12
- Tags: LLM, AI, Production, Machine Learning
- Words: 1306
- Summary: Which LLM application patterns actually survive production, based on extraction, generation and classification features shipped to real users. Includes what to evaluate before building and what reliably fails.


## Introduction

After several years of integrating LLMs into production systems, I've observed a consistent pattern: the features that
deliver real value rarely align with what gets attention at conferences. While the industry focuses on AGI and emergent
behaviors, the mundane applications—data extraction, classification, controlled generation—are quietly transforming how
we build software.

This post presents a framework I've developed for evaluating LLM features based on what actually ships and scales. It's
deliberately narrow in scope, focusing on patterns that have proven reliable across multiple deployments rather than
exploring the theoretical boundaries of what's possible.

For the failure side of this, and why agent chains break even when every individual step looks fine, see [42 Is Not a Random Number](/posts/42-is-not-a-random-number/).

## The Three Categories That Actually Work

Through trial, error, and more error, I've found that LLMs consistently excel in three specific areas. When I'm
evaluating a potential AI feature, I ask: "Does this clearly fit into one of these categories?" If not, it's probably
not worth pursuing (yet).

### 1. Extracting Structured Data from Unstructured Inputs

This is the unsexy workhorse of AI features. Think of it as having an intelligent data entry assistant who never gets
tired of parsing messy inputs.

**What makes this valuable:**

- Humans hate data entry
- Traditional parsing is brittle and breaks with slight format changes
- LLMs can handle ambiguity and variations gracefully

**Real examples I've built:**

- **PDF to JSON converter**: Taking uploaded forms (PDFs, images, even handwritten docs) and extracting structured data.
  What used to require complex OCR pipelines and regex nightmares now works with a simple prompt.
- **API response mapper**: Taking inconsistent third-party API responses and mapping them to your internal data model.
  Every integration engineer's nightmare—different field names, nested structures that change randomly, optional fields
  that are sometimes null and sometimes missing entirely.
- **Customer feedback analyzer**: Extracting actionable insights from the stream of unstructured feedback across emails,
  Slack, support tickets. Automatically pulling out feature requests, bug reports, severity, and sentiment. What used to
  be a PM's full-time job.

The key insight here is that LLMs excel at handling structural variance and ambiguity—the exact things that make
traditional parsers brittle. A single well-crafted prompt can replace hundreds of lines of mapping logic, regex
patterns, and edge case handling. The model's ability to understand intent rather than just pattern match is what makes
this category so powerful.

**Production considerations:** For high-volume extraction from standardized formats, purpose-built services
like [Reducto](https://reducto.ai/) offer better economics and reliability than raw LLM calls. These platforms have
already solved for edge cases around OCR quality, table extraction, and format variations. The build-vs-buy calculation
here typically favors buying unless you have unique requirements or scale that justifies the engineering investment.

### 2. Content Generation and Summarization

This is probably what most people think of when they hear "AI features," but the key is being specific about the use
case.

**What makes this valuable:**

- Reduces cognitive load on users
- Provides consistent quality and tone
- Can process and synthesize large amounts of information quickly

**Real examples I've built:**

- **Smart report generation**: Taking raw data and generating human-readable reports with insights and recommendations.
- **Meeting summarizer**: Processing transcripts to extract key decisions, action items, and important discussions.
- **Documentation assistant**: Generating first drafts of technical documentation from code comments and README files.

The critical lesson here is that unconstrained generation is rarely what you want in production. Effective generation
features require explicit boundaries: output structure, length constraints, tone guidelines, and forbidden topics. The
challenge isn't getting the model to generate—it's getting it to generate within your specific constraints reliably.

This is where prompt engineering transitions from art to engineering: defining schemas, enforcing structural
requirements, and building validation layers. The most successful generation features I've seen treat the LLM as one
component in a larger pipeline, not a magic box.

### 3. Categorization and Classification

This is where LLMs really shine compared to traditional ML. What used to require thousands of labeled examples and
complex training pipelines can now be done with a well-crafted prompt.

**What makes this valuable:**

- No need for labeled training data
- Can handle edge cases and ambiguity
- Easy to adjust categories without retraining

The architectural advantage here is profound: you're essentially defining classifiers declaratively rather than
imperatively. No training data, no model selection, no hyperparameter tuning—just clear descriptions of your categories.
The model's pre-trained understanding of language and context does the heavy lifting.

This fundamentally changes the iteration cycle. Adding a new category or adjusting definitions happens in minutes, not
weeks. The trade-off is less fine-grained control over the decision boundary, but for most business applications, this
is a feature, not a bug.

That said, "just use an LLM" isn't automatically the right call for classification. I ran 32 experiments pitting small
LLMs against a fine-tuned BERT encoder — [the results](/posts/beatingbert/) are a useful counterweight to this section
when latency and cost matter.

**Scaling considerations:** Production deployments require:

- **Structured output guarantees**: Libraries like [Pydantic AI](https://github.com/pydantic/pydantic-ai)
  and [Outlines](https://github.com/outlines-dev/outlines) enforce schema compliance at the token generation level,
  eliminating post-processing failures.
- **Prompt optimization**: [DSPy](https://github.com/stanfordnlp/dspy) and similar frameworks apply optimization
  techniques to prompt engineering, treating it as a learnable parameter rather than a manual craft.
- **Evals, Observability, and Error Analysis**: This could and will likely eventually be its own post

## The Anti-Patterns: What Doesn't Work

Let me save you some pain by sharing what consistently fails:

### 1. Trying to Replace Domain Expertise

LLMs are great at general knowledge but terrible at specialized domains without extensive context. If you need deep
expertise, you still need experts.

### 2. Real-time, High-frequency Operations

Sub-100ms response times and high-frequency calls remain outside the practical envelope for LLM applications. The
latency floor of current models, even with optimizations like speculative decoding, makes them unsuitable for hot-path
operations.

### 3. Anything Requiring Perfect Accuracy

LLMs are probabilistic. If you need 100% accuracy (financial calculations, legal compliance, etc.), use traditional
code.

## A Practical Evaluation Framework

When someone comes to me with an AI feature idea, here's my checklist:

| Question                                     | Good Sign                | Red Flag                    |
|----------------------------------------------|--------------------------|-----------------------------|
| **Does it fit one of the three categories?** | Clear fit with examples  | "It's like ChatGPT but..."  |
| **What's the failure mode?**                 | Graceful degradation     | Catastrophic failure        |
| **Can a human do it in <5 minutes?**         | Yes, but it's tedious    | No, requires deep expertise |
| **Is accuracy critical?**                    | Good enough is fine      | Must be 100% correct        |
| **What's the response time requirement?**    | Seconds are fine         | Needs to be instant         |
| **Do we have the data?**                     | Yes, and it's accessible | "We'll figure it out"       |

## Implementation Strategy

For teams evaluating their first LLM feature, I recommend starting with categorization. The reasoning is purely
pragmatic: it has the clearest evaluation metrics, the most forgiving failure modes, and provides immediate value. You
can validate the approach with a small dataset and scale incrementally.

The implementation complexity is also minimal—you're essentially building a discriminator rather than a generator, which
sidesteps many of the challenges around hallucination, output formatting, and content safety. Most importantly, when
classification confidence is low, you can gracefully fall back to human review without breaking the user experience.

## The Reality of Production AI

The gap between AI demos and production systems remains vast. The features that succeed in production share a common
trait: they augment existing workflows rather than attempting to replace them entirely. They handle the tedious,
error-prone tasks that humans perform inconsistently, freeing cognitive capacity for higher-value work.

This isn't a limitation—it's the current sweet spot for LLM applications. The technology excels at tasks that are
simultaneously too complex for traditional automation but too mundane to justify human attention. Understanding this
paradox is key to building AI features that actually ship.

## Related reading

- [Beating BERT?](/posts/beatingbert/) — 32 experiments on whether small LLMs actually beat a fine-tuned encoder at classification.
- [RAG: From Context Injection to Knowledge Integration](/posts/rag/) — the architectural limits of the most common LLM feature pattern.
- [The Case Against pgvector](/posts/the-case-against-pgvector/) — what the retrieval layer costs you once it's in production.


================================================================================

# A Computer Made This

- URL: https://alex-jacobs.com/posts/notacomputer/
- Author: Alex Jacobs
- Published: 2025-03-27
- Last updated: 2026-10-12
- Tags: AI, Image Generation, OpenAI
- Words: 516
- Summary: OpenAI's 4o generates images by reasoning in pixel space rather than diffusing noise, which is why it can render legible text and follow multi-step edits. What that shifts for creative work.



The past 24 hours have had me navigating an existential crisis while simultaneously being gaslit by friends, family, and colleagues about what's going on. And that's probably fair of them—I have a tendency to overreact to things, to be a bit dramatic.

I am 100% the guy in this panel right now.

![A 4o-generated image opening the post, used as the example under discussion.](img1.jpeg)

But 4o image generation is insane.

I've been working in the LLM space since before ChatGPT shifted everything. I've closely followed the progress. I test every new release. I tell my friends that every AI app they send me is slop. I am not easily impressed.

But this feels like another ChatGPT moment. This isn't just better distribution (is hiding your state-of-the-art model in a Discord chat behind /commands really the best way to get people to use it?). 

This feels foundational. It's not just a better diffusion model—it's actual reasoning in pixel space.

I've been on the fence about whether AGI (whatever that even means) is possible. Can we actually bottle intelligence into an electric rock? But it doesn't take much napkin math to pencil this out a few years. (True believers might ask where I've been, but rest easy brethren—I am yours now.)

It brings to mind this 100% real needlepoint of an Ilya Sutskever quote.
![A 4o-generated scene showing consistent lighting and object placement across a multi-step edit.](img_2.png)

Trying to game out the second- and third-order effects of an image generation model feels strange, even dumb. Infinite Ghibli? What are you worried about? Ghibli gonna take all the jobs? 

If I'm a graphic designer, it is over for me today. But I'm not, I'm a software engineer my job is safe!  I have felt like chicken little screaming into the void about the computers coming for a few years now.  Today I feel some combination of awe and dread. (This probably ends with [most of us becoming electricians](https://intercoast.edu/articles/the-looming-electrician-shortage-how-it-could-impact-americas-economy-and-the-future-of-ai/) so we can wire up the data centers)

I won't even try to get into what the post-reality-filter stage of society we're about to enter looks like, or what this will do to the meme economy.  (My parents already can't tell the difference between AI-generated and real images.  Maybe I can't either.)

I think this tweet (shared with me by a friend this morning) just about sums it up. 
![A 4o-generated image demonstrating accurate in-image text rendering.](img.png)

But to that, I say:
![A follow-up 4o generation keeping the same subject and style after an edit instruction.](img_1.png)

Below is a series I've been working on to try and demonstrate this phenomenon I'm experiencing. My fiancée and I got engaged last October, and we captured an amazing photo (maybe my favorite picture ever). So I've been trying to recreate it in every style possible.

The consistency of this model is incredible--and the content filters are tuned to low right now.  I won't be surprised if by the time you're 
reading this, most of these styles will be blocked (already seems to be happening :/ )

Our original photo:
![A reference photograph used as the input image for the generation that follows.](IMG_1156.jpg)

Lego:
![A 4o-generated restyling of the reference photograph.](img_3.png)
Claymation:
![A second 4o restyling of the same reference photograph in a different visual style.](img_4.png)
Sesame Street:
![A third 4o restyling of the reference photograph, showing style transfer while preserving the subject.](img_5.png)
Scooby-Doo:
![The reference photograph regenerated by 4o in a Scooby-Doo style.](img_10.png)
Neon Sign:
![The reference photograph regenerated by 4o in a Neon Sign style.](img_11.png)
Tim Burton:
![The reference photograph regenerated by 4o in a Tim Burton style.](img_12.png)
Hey Arnold:
![The reference photograph regenerated by 4o in a Hey Arnold style.](img_13.png)
Victorian Botanical Print:
![The reference photograph regenerated by 4o in a Victorian Botanical Print style.](img_14.png)
Wes Anderson:
![The reference photograph regenerated by 4o in a Wes Anderson style.](img_7.png)
Pixar:
![The reference photograph regenerated by 4o in a Pixar style.](img_8.png)
Vintage Comic:
![The reference photograph regenerated by 4o in a Vintage Comic style.](img_9.png)
Peanuts:
![The reference photograph regenerated by 4o in a Peanuts style.](img_6.png)
"Yellow Submarine Family":
![The reference photograph regenerated by 4o in a Yellow Submarine Family style.](img_16.png)
Construction Paper:
![The reference photograph regenerated by 4o in a Construction Paper style.](img_17.png)
Architectural Blueprint:
![The reference photograph regenerated by 4o in a Architectural Blueprint style.](img_18.png)
Medieval Manuscript:
![The reference photograph regenerated by 4o in a Medieval Manuscript style.](medival_manuscript.png)
Street Art Stencil:
![The reference photograph regenerated by 4o in a Street Art Stencil style.](street-art-stencil.png)
Pixelated Video Game:
![The reference photograph regenerated by 4o in a Pixelated Video Game style.](pixelated-video-game.png)
1960s Style Cartoon:
![The reference photograph regenerated by 4o in a 1960s Style Cartoon style.](1960s-style-cartoon.png)
Stop Motion:
![The reference photograph regenerated by 4o in a Stop Motion style.](stop-motion.png)

And finally, Ghibli:
![The reference photograph regenerated by 4o in a And finally, Ghibli style.](img_19.png)

Other models could already do this!

No, they couldn't.

================================================================================

# RAG: From Context Injection to Knowledge Integration

- URL: https://alex-jacobs.com/posts/rag/
- Author: Alex Jacobs
- Published: 2025-02-17
- Last updated: 2026-10-12
- Tags: RAG, LLM, AI, Architecture
- Words: 934
- Summary: A technical dive into the limitations of current RAG approaches, examining architectural challenges and exploring pathways to more integrated knowledge-aware LLM architectures.



## Retrieval-Augmented Generation: Architectural Limitations and Future Directions

Retrieval-Augmented Generation (RAG) has rapidly become a cornerstone in the practical application of Large Language Models (LLMs). Its promise is compelling: to expand LLMs beyond their training data by connecting them to external knowledge sources – from enterprise databases and real-time data streams to proprietary knowledge bases. The allure of RAG lies in its apparent simplicity – augment the LLM's input context with retrieved information, and witness enhanced output quality. However, beneath this layer of simplicity lies a more complex reality--its a bit of a hack. RAG only works because LLMs are generally robust. The more you think on it, the more it becomes clear it _shouldn't_ really work, and should serve only as a stepping stone to a new paradigm.

## Generation vs. Retrieval

At their core, LLMs are generative models that produce text by navigating through a high-dimensional latent space. During pre-training on large datasets, these models learn to map language into this space, capturing relationships between words, phrases, and concepts. Text generation isn't a simple lookup process - it's a sequential operation where the model predicts each token based on both the previous context and its learned representations.

RAG changes this core process significantly. Rather than relying only on the model's learned representations, RAG injects external information directly into the context window alongside the user's query. While this works well in practice, it raises important questions about the theoretical and architectural implications:

1. **Impact on Generation Quality:** How does inserting external information affect the model's learned generation process? Does mixing training-derived and retrieved information create inconsistencies in the model's outputs?

2. **Information Integration:** Can the model effectively combine information from different sources during generation? Or is it simply stitching together pieces without truly understanding how they relate?

3. **Architectural Fitness:** Are transformer architectures and their training objectives actually suited for combining retrieved information with generation? Or are we forcing an approach that doesn't align with how these models were designed to work?

## Real-World Limitations

These theoretical concerns manifest in several practical ways:

### 1. Context Integration Problems

Current RAG implementations often struggle with:
- Abrupt transitions between retrieved content and generated text
- Inconsistent voice and style when mixing sources
- Difficulty maintaining coherent reasoning across retrieved facts
- Limited ability to synthesize information from multiple sources

### 2. Attention Mechanism Overload

The transformer's attention mechanism faces significant challenges:
- Managing attention across disconnected chunks of information
- Balancing focus between query, retrieved content, and generated text
- Handling potentially contradictory information from different sources
- Maintaining coherence when dealing with multiple retrieved documents

### 3. Knowledge Conflicts

RAG systems often struggle to resolve conflicts between:
- The model's pretrained knowledge
- Retrieved information
- Different retrieved sources
- User queries and retrieved content

## The Path Forward: Beyond Basic RAG

Recent research and development suggest several promising directions for addressing these limitations:

### 1. Improved Knowledge Integration

Future systems might:
- Process retrieved information before injection
- Maintain explicit source tracking throughout generation
- Use structured knowledge representations
- Implement hierarchical attention mechanisms

### 2. Enhanced Source Handling

Advanced approaches could:
- Evaluate source reliability and relevance
- Resolve conflicts between sources
- Maintain provenance information
- Generate explicit citations and references

### 3. Architectural Innovations

New architectures might include:
- Dedicated pathways for retrieved information
- Specialized attention mechanisms for source integration
- Dynamic context window management
- Explicit fact-checking mechanisms

## The Next Evolution: Anthropic's Citations API

Anthropic's Citations API represents a significant step beyond traditional RAG implementations. While the exact implementation details aren't public, we can make informed speculations about its architectural innovations based on the capabilities it demonstrates.

### Architectural Innovations

The Citations API likely goes beyond simple prompt engineering to include fundamental architectural changes:

1. **Enhanced Context Processing**
   - Specialized attention mechanisms for source document processing
   - Dedicated layers for maintaining source awareness throughout generation
   - Architectural separation between query processing and source document handling
   - Advanced chunking and document representation strategies

2. **Citation-Aware Generation**
   - Built-in tracking of source-claim relationships
   - Automatic detection of when citations are needed
   - Dynamic weighting of source relevance
   - Real-time fact verification against sources

3. **Training Innovations**
   - Custom loss functions for citation accuracy
   - Source fidelity metrics during training
   - Explicit training for source grounding
   - Specialized datasets for citation learning

### Speculation on Implementation

The system likely employs several key mechanisms:

1. **Dual-Stream Processing**
   - Separate processing paths for user queries and source documents
   - Specialized attention heads for citation tracking
   - Fusion layers for combining information streams
   - Dynamic context management

2. **Source Integration**
   - Fine-grained document chunking
   - Semantic similarity tracking
   - Citation boundary detection
   - Provenance preservation

3. **Training Approach**
   - Multi-task training combining generation and citation
   - Custom datasets focused on source grounding
   - Citation-specific loss functions
   - Source fidelity metrics

## Beyond Traditional RAG

The Citations API and similar emerging technologies point to a future where knowledge integration isn't just an add-on but a core capability of language models. This evolution requires moving beyond simply stuffing context windows with retrieved documents toward architectures specifically designed for knowledge-aware generation.

The next generation of these systems will likely feature:
- Native citation capabilities
- Real-time fact verification
- Seamless source integration
- Dynamic knowledge updates
- Explicit handling of source conflicts

As we move forward, the goal isn't to patch the limitations of current RAG systems but to fundamentally rethink how we combine language models with external knowledge. This might lead to entirely new architectures specifically designed for knowledge-enhanced generation, moving us beyond the current paradigm of context window injection toward truly integrated knowledge-aware AI systems.

## Related reading

- [CheeseGPT](/posts/cheesegpt/) — a simple, end-to-end RAG system built from scratch, if you want the concrete version of everything above.
- [The Case Against pgvector](/posts/the-case-against-pgvector/) — the retrieval half of RAG has to run somewhere, and the storage layer has sharper edges in production than most tutorials admit.
- [A Production Framework for LLM Feature Evaluation](/posts/practicalaifeatures/) — which LLM features actually survive contact with production.

================================================================================

# Python Async Programming: A Deep Dive into async/await and the Event Loop

- URL: https://alex-jacobs.com/posts/pythonasync/
- Author: Alex Jacobs
- Published: 2024-01-28
- Last updated: 2026-08-20
- Tags: Python, Async, Backend
- Words: 3756
- Summary: How async Python actually works: what async def and await do, how the asyncio event loop schedules coroutines, tasks and futures explained, and when to use async instead of threads or processes.


## The short version

If you just want the mental model:

- **`async def` defines a coroutine function.** Calling it doesn't run anything — it returns a coroutine object, which is a description of work, not the work itself.
- **`await` pauses the current coroutine** and hands control back to the event loop, which is then free to run something else until the awaited thing finishes.
- **The event loop is a single-threaded scheduler.** There is no parallelism in async Python. It's very efficient turn-taking, and the turns only change at an `await`.
- **Async helps when you're I/O-bound** — waiting on networks, APIs, databases, disks. It does nothing for CPU-bound work; that's what `multiprocessing` is for.
- **One blocking call stalls everything.** A `time.sleep()` or a synchronous DB driver anywhere in a coroutine freezes the entire loop, not just that coroutine.

The rest of this post is about why each of those is true, and what's happening underneath.

## Why async Python exists

Modern applications — especially anything dealing with network requests or many concurrent operations — need to handle multiple things at once. For a long time, threading and multiprocessing were the go-to answers in Python. But the Global Interpreter Lock and the overhead of thread management limit how well those approaches work, particularly for I/O-bound workloads where the CPU is mostly sitting idle waiting on something external.

Async is the alternative. The surface syntax of `async` and `await` looks simple enough, but understanding what's happening beneath it is what separates working async code from code that mysteriously runs no faster than the synchronous version. We'll cover the event loop, coroutines, tasks, futures, context switching, and how async compares to threads and processes.

## `async def` and `await`: the syntax

These two keywords are the building blocks for writing asynchronous code in Python.

### What `async def` actually does

In Python, you declare a function as asynchronous by using the `async def` syntax instead of the regular `def`.  This seemingly small change has profound implications.  An `async def` function, also known as a **coroutine function**, doesn't execute like a regular synchronous function. Instead, it returns a **coroutine object**.

Think of a coroutine object as a promise of work to be done later. It's not the work itself, but rather a representation of that work, ready to be executed when the time is right.

```python
import asyncio

async def hello_async():
    print("Hello from async function!")
    await asyncio.sleep(1) # Simulate some async operation (like network I/O)
    print("Async function finished.")

# Calling the async function returns a coroutine object
coro = hello_async()
print(f"Coroutine object: {coro}")

# To actually run the coroutine, we need to use an event loop
asyncio.run(coro)
```

In this example, `hello_async` is an asynchronous function.  When we call `hello_async()`, it doesn't immediately print "Hello from async function!". Instead, it creates and returns a coroutine object. To actually execute the code within the coroutine, we need to use `asyncio.run()`, which sets up and runs an **event loop** (more on event loops shortly) to manage the execution of our coroutine.

### What `await` actually does

The `await` keyword is the other half of the async duo.  It can only be used inside `async def` functions.  `await` is the point where an asynchronous function can **pause** its execution and yield control back to the event loop.  This is the crucial mechanism that enables concurrency in async Python without relying on threads.

When you `await` something, you are essentially saying: "I need to wait for this asynchronous operation to complete.  While I'm waiting, I'm going to yield control back to the event loop so it can work on other tasks. Once this operation is done, please resume my execution from right here."

In our `hello_async` example, `await asyncio.sleep(1)` is a simulated asynchronous operation that represents waiting for 1 second.  During this second, the `hello_async` coroutine pauses, and the event loop is free to execute other coroutines or handle other events.  Once the sleep duration is over, the event loop resumes the `hello_async` coroutine from the line after the `await` statement, and it prints "Async function finished."

**Important Note:**  You can only `await` objects that are **awaitable**.  In practice, this usually means you're awaiting other coroutines, `Future` objects (which we'll discuss later), or objects that have implemented the `__await__` special method.  Standard synchronous functions are *not* awaitable.

### A complete async Python example

Here's the thing the syntax alone doesn't show you: `await` on its own doesn't buy you any concurrency. If you `await` three things one after another, you wait for all three in sequence — exactly like synchronous code, just with extra keywords.

```python
import asyncio
import time

async def fetch(name, seconds):
    print(f"{name}: starting")
    await asyncio.sleep(seconds)   # stands in for a network call
    print(f"{name}: done")
    return f"{name} result"

async def sequential():
    start = time.perf_counter()
    await fetch("A", 1)
    await fetch("B", 1)
    await fetch("C", 1)
    print(f"sequential took {time.perf_counter() - start:.1f}s")

async def concurrent():
    start = time.perf_counter()
    results = await asyncio.gather(
        fetch("A", 1),
        fetch("B", 1),
        fetch("C", 1),
    )
    print(f"concurrent took {time.perf_counter() - start:.1f}s")
    return results

asyncio.run(sequential())   # sequential took 3.0s
asyncio.run(concurrent())   # concurrent took 1.0s
```

Same coroutines, same `await`, 3x difference. The concurrency comes from `asyncio.gather()` handing the event loop three coroutines at once, so it can start all three and then sit on the `await` while every one of them is in flight. In the sequential version the loop only ever knows about one at a time.

This is the single most common mistake in async Python: writing `await` in a loop and wondering why nothing got faster.

```python
# Slow — one at a time, despite being "async"
results = []
for url in urls:
    results.append(await fetch_url(url))

# Fast — all in flight at once
results = await asyncio.gather(*(fetch_url(url) for url in urls))
```

One more trap worth knowing early. Because the event loop is single-threaded, a *blocking* call inside a coroutine doesn't just block that coroutine, it blocks everything:

```python
async def bad():
    time.sleep(1)          # blocks the entire event loop for a full second
    
async def good():
    await asyncio.sleep(1) # yields control; other coroutines run
```

The same applies to any synchronous library — a non-async database driver, `requests` instead of `aiohttp`, a heavy CPU computation. If you must call blocking code, push it off the loop with `asyncio.to_thread()`.

## The asyncio event loop

At the heart of async Python lies the **event loop**.  Think of the event loop as the central conductor of an orchestra.  It's responsible for managing and scheduling the execution of all your asynchronous tasks.  It's a single-threaded loop that constantly monitors for events and dispatches tasks to be executed when those events occur.

### How the event loop works

1.  **Task Queue:** The event loop maintains a queue of tasks (usually coroutines wrapped in `Task` objects) that are ready to be executed or resumed.
2.  **Event Monitoring:** The event loop also monitors for various events, such as network sockets becoming ready for reading or writing, timers expiring, or file operations completing.  It typically uses efficient system calls like `select`, `poll`, or `epoll` (depending on the operating system) to monitor these events without blocking.
3.  **Task Execution and Resumption:** When an event occurs that makes a task ready to proceed (e.g., data is available on a socket that a task is waiting to read from), the event loop picks up that task from the queue and executes it until it encounters an `await` statement.
4.  **Yielding Control with `await`:** When a coroutine reaches an `await` statement, it effectively tells the event loop, "I need to wait for this operation.  Please pause me and let someone else run."  The event loop then takes control and looks for other tasks in the queue that are ready to run.
5.  **Resuming Execution:** Once the awaited operation completes (e.g., the network request returns, the timer expires), the event loop is notified.  It then puts the paused coroutine back into the task queue, ready to be resumed at the point where it left off.
6.  **Looping Continuously:** The event loop continues this process of monitoring events, executing tasks, and pausing/resuming coroutines in a loop until there are no more tasks to run or the program is explicitly stopped.

### asyncio, uvloop, and other event loop implementations

Python's standard library provides the `asyncio` module, which includes a built-in event loop implementation. This default event loop is written in Python and is generally sufficient for many use cases.

However, for performance-critical applications, especially those dealing with high-performance networking, you might consider using alternative event loop implementations.  One popular option is **`uvloop`**.

**`uvloop`**: `uvloop` is a blazing-fast, drop-in replacement for `asyncio`'s event loop. It's written in Cython and built on top of `libuv`, the same high-performance library that powers Node.js.  `uvloop` is significantly faster than the default `asyncio` event loop, especially for network I/O.

To use `uvloop`, you typically need to install it separately (`pip install uvloop`) and then set it as the event loop policy for `asyncio` when your application starts:

```python
import asyncio
import uvloop

async def main():
    print("Running with uvloop!")
    await asyncio.sleep(1)
    print("Done.")

if __name__ == "__main__":
    uvloop.install() # Set uvloop as the event loop policy
    asyncio.run(main())
```

Other event loop implementations exist, but `asyncio` and `uvloop` are the most commonly used in Python async programming.  Choosing between them often depends on the performance requirements of your application. For most general async tasks, `asyncio`'s default loop is perfectly adequate.  For high-load network applications, `uvloop` can provide a noticeable performance boost.

## Coroutines, tasks, and futures

To truly understand async Python, it's helpful to think about it in layers.  We've already touched upon coroutines and the event loop. Let's now delve into the roles of **Tasks** and **Futures**.

### Coroutines

As we discussed earlier, coroutines are the asynchronous functions you define using `async def`. They represent units of asynchronous work.  Coroutines themselves are not directly executed by the event loop. Instead, they need to be wrapped in something that the event loop can manage and schedule.  This "something" is a **Task**.

### Tasks: running coroutines concurrently

A **Task** in `asyncio` is essentially a wrapper around a coroutine that allows the event loop to schedule and manage its execution.  When you want to run a coroutine concurrently within the event loop, you typically create a Task from it.

You can create a Task using `asyncio.create_task()`:

```python
import asyncio

async def my_coroutine():
    print("Coroutine started")
    await asyncio.sleep(2)
    print("Coroutine finished")
    return "Coroutine result"

async def main():
    task1 = asyncio.create_task(my_coroutine()) # Create a Task from the coroutine
    task2 = asyncio.create_task(my_coroutine()) # Create another Task

    print("Tasks created, waiting for completion...")

    result1 = await task1 # Await the completion of task1
    result2 = await task2 # Await the completion of task2

    print(f"Task 1 result: {result1}")
    print(f"Task 2 result: {result2}")

asyncio.run(main())
```

In this example, we create two Tasks, `task1` and `task2`, from the same `my_coroutine`.  `asyncio.create_task()` schedules these coroutines to be run by the event loop concurrently. When we `await task1` and `await task2`, we are waiting for these Tasks to complete and retrieve their results.

Tasks are essential for managing the lifecycle of coroutines within the event loop. They provide methods to:

*   **Cancel a Task:**  `task.cancel()`
*   **Check if a Task is done:** `task.done()`
*   **Get the result of a Task:** `task.result()` (if done)
*   **Get exceptions raised during Task execution:** `task.exception()` (if any)

### Futures: results that haven't arrived yet

A **Future** is an object that represents the eventual result of an asynchronous operation.  It's a placeholder for a value that might not be available yet.  Tasks in `asyncio` are actually a subclass of Futures.

Futures are used extensively in async Python to represent the outcome of operations that are performed asynchronously, such as:

*   **Network I/O:**  Reading data from a socket, sending a request to a server.
*   **File I/O:** Reading or writing to a file (in an async context).
*   **Concurrent computations:**  Tasks running in parallel (within the same event loop or in different threads/processes).

A Future object has a state that can be:

*   **Pending:** The asynchronous operation is still in progress.
*   **Running:**  The operation is currently being executed.
*   **Done:** The operation has completed successfully or with an exception.
*   **Cancelled:** The operation has been cancelled.

You can interact with a Future to:

*   **Check if it's done:** `future.done()`
*   **Get the result:** `future.result()` (blocks until done if pending, raises exception if an exception occurred)
*   **Get exceptions:** `future.exception()` (returns exception if one occurred, otherwise `None`)
*   **Add callbacks:** `future.add_done_callback(callback_function)` (run a function when the future is done)
*   **Cancel the future:** `future.cancel()`

When you `await` a Task (or any awaitable Future-like object), you are essentially waiting for that Future to become "done" and then retrieving its result or handling any exceptions.

## How coroutines pause and resume

Now let's look at how coroutines are actually executed, and how context switching works in async Python.

### Cooperative vs. preemptive multitasking

Async Python uses **cooperative multitasking**.  This is in contrast to **preemptive multitasking** used by operating systems for threads and processes.

*   **Preemptive Multitasking (Threads/Processes):** In preemptive multitasking, the operating system's scheduler decides when to switch between threads or processes.  It can interrupt a running thread/process at any time and switch to another, even if the running thread/process doesn't explicitly yield control.  This is typically based on time slices and priority levels.
*   **Cooperative Multitasking (Async Python):** In cooperative multitasking, coroutines voluntarily yield control back to the event loop when they encounter an `await` statement.  The event loop then decides which coroutine to run next.  Context switching only happens at these explicit `await` points.  A coroutine will continue to run until it reaches an `await` or completes.

This cooperative nature has important implications:

*   **No True Parallelism (within a single event loop):**  Within a single event loop running in a single thread, true parallelism is not achieved.  Coroutines take turns running.  If a coroutine doesn't `await` frequently and performs long-running CPU-bound operations, it can block the event loop and prevent other coroutines from making progress.
*   **Responsiveness:** Cooperative multitasking is excellent for I/O-bound tasks. While one coroutine is waiting for I/O, another can run, keeping the application responsive.
*   **Less Overhead:** Context switching in cooperative multitasking is generally lighter than preemptive context switching between threads or processes. There's less operating system overhead involved.
*   **Deterministic Behavior (mostly):** Because context switching happens only at explicit `await` points, the execution flow of async code is often more predictable and easier to reason about compared to multithreaded code, which can have race conditions and unpredictable scheduling.

### What happens at an `await`

When a coroutine reaches an `await` statement, several things happen:

1.  **`await` Expression:** The expression after `await` (e.g., `asyncio.sleep(1)`, another coroutine, a Future) must be awaitable.
2.  **Yielding Control:** The coroutine effectively "pauses" its execution at the `await` point. It returns control back to the event loop.
3.  **Event Loop Takes Over:** The event loop becomes active again. It looks at its task queue for other tasks that are ready to run.
4.  **Registering for Resumption:**  The coroutine, along with information about where it paused (the line after the `await`), is registered with the event loop as being "waiting" for the completion of the awaited operation.
5.  **Awaited Operation Proceeds:** The awaited operation (e.g., network request, timer) proceeds asynchronously in the background (often managed by non-blocking system calls).
6.  **Event Notification:** When the awaited operation is complete, the event loop receives a notification (e.g., socket becomes readable, timer expires).
7.  **Resuming the Coroutine:** The event loop puts the paused coroutine back into the task queue, marked as ready to be resumed.
8.  **Coroutine Resumes:** When the event loop gets around to executing this coroutine again, it resumes from the exact point where it was paused (right after the `await` statement).  It now has access to the result of the awaited operation (if any).

This pause-and-resume mechanism is what allows asynchronous code to be written in a seemingly sequential style, even though it's actually being executed in an interleaved and non-blocking manner.

## Async vs. threads vs. multiprocessing in Python

It's worth being clear about when async is the right choice and when threading or multiprocessing is the better fit. They solve different problems, and reaching for the wrong one is how you end up with async code that runs no faster than the version you replaced.

### When to use async: I/O-bound work

Async Python excels in scenarios where your application is **I/O-bound**.  This means that the primary bottleneck is waiting for external operations to complete, such as:

*   **Network requests:**  Fetching data from APIs, making HTTP requests, communicating with databases over a network.
*   **File I/O:**  Reading and writing to files (especially over a network file system).
*   **Waiting for user input:**  In GUI applications or interactive systems.

In these cases, the CPU is often idle while waiting for I/O operations. Async Python allows you to utilize this idle time by letting other coroutines run while one is waiting for I/O.  It's highly efficient for handling many concurrent I/O operations with minimal overhead.

**Example: Web Server**

A web server that handles many concurrent requests is a classic example where async Python shines.  While one request is being processed (which often involves waiting for database queries, external API calls, etc.), the server can be handling other requests concurrently.

```python
import asyncio
import aiohttp
from aiohttp import web

async def fetch_data_from_api(api_url):
    async with aiohttp.ClientSession() as session:
        async with session.get(api_url) as response:
            return await response.json()

async def handler(request):
    data = await fetch_data_from_api("https://api.example.com/data")
    return web.json_response(data)

async def main():
    app = web.Application()
    app.add_routes([web.get('/', handler)])
    runner = web.AppRunner(app)
    await runner.setup()
    site = web.TCPSite(runner, 'localhost', 8080)
    await site.start()
    print("Server started at http://localhost:8080")
    await asyncio.Event().wait() # Keep the server running indefinitely

if __name__ == "__main__":
    asyncio.run(main())
```

This example uses `aiohttp`, an async HTTP client and server library.  The `fetch_data_from_api` coroutine performs an asynchronous HTTP request.  The `handler` coroutine uses `await fetch_data_from_api` to fetch data without blocking the server.  The server can handle many requests concurrently, making it highly scalable for I/O-bound web applications.

### When to use threads

Threads, especially when used with Python's `threading` module, are suitable for tasks that are more **CPU-bound** and can benefit from **concurrency** (even if not true parallelism due to the GIL).

*   **CPU-Bound Tasks:** Tasks that spend most of their time performing computations on the CPU, rather than waiting for I/O.  Examples include:
    *   Image processing
    *   Numerical computations
    *   Data analysis
    *   Cryptographic operations

*   **Concurrency (with GIL limitations):** Python's GIL (Global Interpreter Lock) prevents true parallelism for CPU-bound tasks in standard CPython threads. Only one thread can hold the Python interpreter lock at any given time.  However, threads can still provide concurrency by releasing the GIL during I/O operations or certain blocking system calls.  This can improve responsiveness even for CPU-bound tasks if they involve some I/O or blocking.

**Example: CPU-Bound Computation (with threading for concurrency)**

```python
import threading
import time

def cpu_bound_task(task_id):
    print(f"Task {task_id} started")
    start_time = time.time()
    result = 0
    for _ in range(10**7): # Simulate CPU-intensive computation
        result += 1
    end_time = time.time()
    duration = end_time - start_time
    print(f"Task {task_id} finished in {duration:.4f} seconds, result: {result}")

def main_threads():
    threads = []
    for i in range(4):
        thread = threading.Thread(target=cpu_bound_task, args=(i,))
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join() # Wait for all threads to complete

if __name__ == "__main__":
    main_threads()
```

In this example, `cpu_bound_task` simulates a CPU-intensive operation.  We create multiple threads to run this task concurrently.  While the GIL limits true parallelism for CPU-bound Python code, threads can still provide some concurrency and potential performance improvement, especially if the tasks involve some I/O or blocking operations.  For purely CPU-bound tasks, however, the benefits might be limited by the GIL.

### When to use processes: true parallelism

For truly **CPU-bound and computationally intensive tasks** that need **true parallelism** and to bypass the GIL limitations, **multiprocessing** using Python's `multiprocessing` module is the way to go.

*   **True Parallelism:** Multiprocessing creates separate processes, each with its own Python interpreter and memory space. Processes run in parallel on multiple CPU cores, achieving true parallelism for CPU-bound tasks.
*   **Bypassing the GIL:** Each process has its own GIL, so the GIL limitation of threads is overcome.
*   **Higher Overhead:** Process creation and inter-process communication have more overhead compared to threads or async tasks. Processes consume more system resources (memory, process management overhead).

**Example: CPU-Bound Computation (with multiprocessing for parallelism)**

```python
import multiprocessing
import time

def cpu_bound_task_process(task_id): # Same CPU-bound task as before
    print(f"Process {task_id} started")
    start_time = time.time()
    result = 0
    for _ in range(10**7):
        result += 1
    end_time = time.time()
    duration = end_time - start_time
    print(f"Process {task_id} finished in {duration:.4f} seconds, result: {result}")

def main_processes():
    processes = []
    for i in range(4):
        process = multiprocessing.Process(target=cpu_bound_task_process, args=(i,))
        processes.append(process)
        process.start()

    for process in processes:
        process.join() # Wait for all processes to complete

if __name__ == "__main__":
    main_processes()
```

In this multiprocessing example, we create separate processes to run the same CPU-bound task.  Because each process has its own interpreter and bypasses the GIL, we can achieve true parallelism and significantly speed up CPU-intensive computations on multi-core systems.

### Choosing between async, threads, and processes

| Feature                | Async Python (asyncio) | Threads (threading) | Processes (multiprocessing) |
| ---------------------- | ----------------------- | -------------------- | --------------------------- |
| **Task Type**          | I/O-bound               | CPU-bound (with I/O) | CPU-bound (pure computation)|
| **Parallelism**        | No (within loop)        | Limited (due to GIL) | Yes (true parallelism)      |
| **Concurrency**        | High                    | Moderate             | High                        |
| **Overhead**           | Low                     | Moderate             | High                        |
| **GIL Impact**         | Not affected            | Limited by GIL       | Bypasses GIL                |
| **Context Switching**  | Cooperative (light)    | Preemptive (OS)      | Preemptive (OS)             |
| **Memory Footprint**   | Lower                   | Moderate             | Higher                      |
| **Complexity**         | Moderate                | Moderate             | Higher (IPC needed)         |
| **Use Cases**          | Web servers, network apps, UI | Concurrent I/O + CPU | CPU-intensive computations  |

**General Guidelines:**

*   **I/O-Bound, High Concurrency:** Async Python (asyncio) is often the best choice.
*   **CPU-Bound with some I/O, Responsiveness:** Threads (threading) can be considered, but be mindful of GIL limitations for pure CPU-bound tasks.
*   **CPU-Bound, True Parallelism, Max Performance:** Processes (multiprocessing) are essential, especially for computationally intensive tasks on multi-core machines.
*   **Hybrid Applications:** You can combine async and multiprocessing. For example, use async for handling network I/O and multiprocessing for CPU-bound background tasks.

## When to reach for async

Async Python is a good way to write concurrent code for I/O-bound applications, and understanding the machinery underneath — the event loop, coroutines, tasks, futures, cooperative multitasking — is what makes the difference between using it well and using it superstitiously.

It isn't a silver bullet. It won't speed up CPU-bound work, it doesn't replace threading or multiprocessing, and a single blocking call will undo all of it. But for the case it's designed for — lots of concurrent waiting on things outside your process — nothing else in Python comes close on overhead.

If you take three things from this post: `async def` returns a coroutine rather than running one, concurrency comes from `gather` and tasks rather than from `await` itself, and the loop is single-threaded so anything that blocks blocks everyone.

If you want to see these concepts in a real application, FastAPI is async Python end to end — I've written about [integration testing FastAPI](/posts/fastapitests/), where the event loop and `async def` shape how you have to structure your test fixtures.

================================================================================

# Integration Testing FastAPI: Mocking Auth, MongoDB, S3, and External APIs

- URL: https://alex-jacobs.com/posts/fastapitests/
- Author: Alex Jacobs
- Published: 2023-06-11
- Last updated: 2026-08-20
- Tags: Python, FastAPI, Testing, Backend
- Words: 3465
- Summary: A practical guide to FastAPI integration tests with pytest: mocking JWT authentication and the TestClient, patching external API calls, and faking MongoDB and S3 with mongomock and moto.


Follow along with all the code [here](https://github.com/alexjacobs08/fastApi-Integration-tests)

## Introduction

If you're a backend developer working with FastAPI, you already know the framework excels at simplifying API development
with features like async support and automated Swagger docs. However, when it comes to integration testing--particularly
mocking external services like MongoDB, AWS S3, and third-party APIs—the waters can get murky. This post is your guide
to navigating these complexities.

This isn't a one-stop-shop for all things testing; the focus is squarely on integration testing within
FastAPI.

**Prerequisites:** Familiarity with FastAPI, PyTest, MongoDB, and AWS S3 is assumed. If you're new to any of these
technologies, you may want to get up to speed before proceeding. FastAPI is built on async Python, and a lot of testing
confusion traces back to not knowing what `async def` and the event loop are really doing — if that's shaky ground,
my [deep dive into Python async programming](/posts/pythonasync/) covers it.

### What is Integration Testing?

Integration testing involves combining individual units of code and testing them as a group. This type of testing aims
to expose faults in the interactions between integrated units. In our context, we could also probably call these
API tests (and you'll see that's how I name my test files). In the context of FastAPI, these units often involve your
API endpoints, external databases like MongoDB, and other services such as AWS S3.

### Why It's Challenging

When compared to unit tests, integration tests in FastAPI present unique difficulties. These challenges mostly stem from
the interaction with external dependencies. Mocking these dependencies is not always straightforward, and mistakes can
lead to false positives or negatives, undermining the purpose of the tests.

### Why It Matters

So, why focus on integration testing in FastAPI? Because it's here that you validate that various services and databases
work in harmony. By ensuring that these integrated units function as expected, you not only increase code reliability
but also save debugging time in the long run.

Unit tests are great for testing individual units of code, but they don't test the interactions between these units.
I also find integration tests (especially this kind, which test our endpoints) are particularly useful before doing a
large refactor--going from a v0 to v1 of an app, where unit tests might not make a ton of sense because you're rewriting
all of your 'units'.

## A Glimpse into Our Test App

Before we delve into the technicalities of mocking various elements, let's take a quick look at the FastAPI application
we'll be using as a test subject. This app serves as a sandbox for our integration tests, built to showcase different
aspects of FastAPI that commonly require mocking in a test environment.

Our sample application has a straightforward schema designed for demonstration purposes:

```yaml
/login  # simple login
  POST: Login

/users/me/  # example for mocking auth
  GET: Read Users Me

/users/me/preferences  # example for mocking mongodB
  GET: Get Preferences
  POST: Save Preferences

/users/me/profile_pic  # example for mocking s3
  GET: Get Profile Picture
  POST: Save Profile Picture

/weather/{city}  # example for mocking external API
  GET: Get Weather
```

Our tests live in `tests/` where they are organized according to the application components they
evaluate. We employ PyTest for test execution and adhere to its conventions for discovering tests and initial set up.
This includes utilizing the `conftest.py` configuration file for shared hooks and fixtures.

## Mocking Authentication in FastAPI

As we move through the article, we'll see that FastAPI provides an easy way to do dependency injection,
which is especially useful for our testing scenarios. Using `dependency_overrides`, we can inject our mocked functions
into the FastAPI app, but remember, this only works for endpoints/functions using the `Depends()` syntax.
You can read more about dependency injection in FastAPI [here](https://fastapi.tiangolo.com/tutorial/dependencies/).

In our FastAPI application, we have endpoints that require authentication. During testing, we don't want to use real
authentication because it would require us to manage real user credentials and tokens. This would complicate our tests
and potentially expose sensitive information. Therefore, we mock the authentication process.

### Testing Our Login

Before we get to dependency injection or mocking out our services, let's make sure our auth works. We have simple
authentication defined in the project that checks if a user is in the database, if their password
matches, and returns a token if they are. (I am not going to focus on how to do authentication, but you can check out
the code in `auth.py` if you're interested.)

For the purposes of this tutorial, our database is just a dictionary with one hard coded user.

###### (I'll be using my name as the username in this tutorial in order to burn it into your brain)

```python
# File: app/db.py
users_db: Dict[str, UserInDB] = {
    "alexjacobs": UserInDB(
        username="alexjacobs",
        email="alex@example.com",
        hashed_password=get_password_hash("secret"),
    )
}
```

and our login endpoint...

```python
# File: app/main.py
@app.post("/login", response_model=Token)
async def login(_login: Login):
    user = authenticate_user(users_db, _login.username, _login.password)
    if not user:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect username or password",
            headers={"WWW-Authenticate": "Bearer"},
        )
    access_token = create_access_token(
        data={"sub": user.username}
    )
    return {"jwt": access_token, "token_type": "bearer"}
```

and we have some simple tests just to verify that our login endpoint works as expected

```python
# File: tests/test_auth_api.py
def test_login(client):
    response = client.post("/login", json={"username": "alexjacobs", "password": "secret"})
    assert response.status_code == 200
    assert "jwt" in response.json()

def test_login_fail(client):
    response = client.post("/login", json={"username": "alexjacobs", "password": "not_the_right_password"})
    assert response.status_code == 401
    assert response.json() == {'detail': 'Incorrect username or password'}
```

These initial tests don't require mocking the authentication process because they are designed to test the
endpoint's basic functionality. They act as a foundational layer upon which we will build more complex test scenarios
that require mocking.

### Mocking Authentication in Our Test Client

Now, let's say we have another endpoint that requires authentication. We'll use the `Depends` function to inject our
authentication function into our endpoint. Our `/users/me` endpoint returns information about the user logged in.

```python
# File: app/main.py
@app.get("/users/me/", response_model=User)
async def read_users_me(_user: TokenData = Depends(get_auth)):
    user = users_db.get(_user.username)
    if user is None:
        raise HTTPException(status_code=404, detail="User not found")
    return user
```

Authentication is handled by the `get_auth` function, which is injected into our endpoint using the `Depends`
function.  
Since we don't want to use real authentication in our tests, we're going to mock this function. I'm going to show
multiple ways of doing this so you can choose the one that works best for you.

The most standard way of mocking this function is to use the `dependency_overrides` feature of FastAPI, before we initialize our
TestClient.

```python
# File: tests/conftest.py
@pytest.fixture
def client(mock_s3_bucket):
    # we patch auth within our client fixture
    from app.main import app

    def mock_get_auth():
        return TokenData(username="alexjacobs")

    app.dependency_overrides[get_auth] = mock_get_auth
    with TestClient(app) as test_client:
        yield test_client

    app.dependency_overrides.clear()
```
You'll see we declare a function called `mock_get_auth` that returns a `TokenData` object.  Then in line 10,
`app.dependency_overrides[get_auth] = mock_get_auth`, we 'inject' our mock function into our app in place of the
`get_auth` function.
 
This code is essentially replacing the `get_auth` function in our app with a mock function that returns the TokenData
object hardcoded into the function we're pathing with.

Then, in our test we pass in the client fixture, and we can see that our test passes.

```python
# File: tests/test_auth_api.py
def test_get_me_patched_auth(client):
    # auth works without real jwt because we patched it in the client fixture
    response = client.get("/users/me", headers={"Authorization": "Bearer " + 'fake.jwt'})
    assert response.status_code == 200
    assert response.json() == {'username': 'alexjacobs', 'email': 'alex@example.com'}
```

(we really don't even need to include auth headers, since we're mocking the auth function, but I'm keeping it
here for clarity)

### Mocking Authentication Directly in your Tests

Now, what if we want to mock the auth function in a different way? We can do this by mocking the auth function directly
in our test.

First, we'll create a second client fixture that doesn't patch the auth function.

```python
# File: tests/conftest.py
@pytest.fixture
def client_unpatched_auth():
    # we don't patch auth for this client
    from app.main import app

    with TestClient(app) as test_client_2:
        yield test_client_2
```

Now we're going to write a test to verify that auth fails (since we're not patching the auth function)

```python
# File: tests/test_auth_api.py
def test_get_me_unpatched_auth(client_unpatched_auth):
    # auth fails without real jwt because we're using the unpatched client fixture
    response = client_unpatched_auth.get("/users/me", headers={"Authorization": "Bearer " + 'fake.jwt'})
    assert response.status_code == 401
    assert response.json() == {"detail": "Could not validate credentials"}
```

To make this work, there are two approaches. The first is to mock the auth function directly in the test

```python
# File: tests/test_auth_api.py
def test_get_me_path_auth_in_fn(client_unpatched_auth):
    # auth works because we patch it in this test (not in the client fixture)
    def mock_get_auth():
        return TokenData(username="alexjacobs")

    app.dependency_overrides[get_auth] = mock_get_auth

    response = client_unpatched_auth.get("/users/me", headers={"Authorization": "Bearer " + 'fake.jwt'})
    assert response.status_code == 200
```

All we've done here is moved the code that patches the auth function into the test itself. This could be useful if you
wanted to mock the auth function in some tests, but not others (but only wanted to have one client fixture).

Next, we'll do the same thing, but we're going to use a new fixture and factory function to create our mock auth function.
So first, we'll create a new fixture,

```python
# File: tests/conftest.py
@pytest.fixture
def mock_get_auth_factory():
    def mock_get_auth():
        return TokenData(username="alexjacobs")

    return mock_get_auth
```

and then we'll use this fixture in our test

```python
# File: tests/test_auth_api.py
def test_get_me_patch_auth_with_fixture(client_unpatched_auth, mock_get_auth_factory):
    # auth works because we patch it with a fixture instead of in the client fixture
    app.dependency_overrides[get_auth] = mock_get_auth_factory
    response = client_unpatched_auth.get("/users/me", headers={"Authorization": "Bearer " + 'fake.jwt'})
    assert response.status_code == 200
```

Notice that when we're passing in the `mock_get_auth_factory` fixture, we're passing in the function itself, not the
return value of the function. This is because the fixture is a factory function that returns a function, so we need to
pass in the function itself.

This may not seem very useful (and for mocking auth, it may not be), but there are plenty of scenarios when you may
want to mock a function in some tests, but not others. This is a good way to do that.

## Mocking External APIs

Our FastAPI application makes external API calls to fetch weather data. During testing, we don't want to make real API
calls because they can be slow and unreliable. Therefore, we mock the external API calls.

We mock the external API calls by patching the `fetch_weather` function in our FastAPI application. This function is
responsible for making the actual API call and returning the weather data. We replace it with a mock function that
always returns a predefined weather data. But, this function does use dependency injection (the `Depends` funciton)
in our app, so we have to mock it in a different way.

First, let's look at our endpoint and the function it calls.

```python
# File: app/main.py
@app.get("/weather/{city}", response_model=WeatherResponse)
async def get_weather(city: str, _user: TokenData = Depends(get_auth)):
    weather = fetch_weather(city)
    return WeatherResponse(city=city, weather=weather)
```

and our `fetch_weather()` function

```python
# File: app/helpers.py
def fetch_weather(city_name):
    url = f"http://wttr.in/{city_name}?format=%C+%t"
    response = requests.get(url)
    return response.text
```

Pretty simple... we are hitting an external API, though, so we need to mock this out in our tests.

We have three ways to mock the external API calls:

### Mocking External APIs with unittest.mock.patch Directly in Our Test

Using unittest.mock.patch, we patch our `fetch_weather` function to return 'sunny'.
We then verify our endpoint returns the expected response (and verify that our mock function was called--this last part 
probably isn't really necessary here, but is a good demonstration)

```python
# File: tests/test_weather_api.py
def test_get_weather(client):
    with patch('app.main.fetch_weather', return_value='sunny') as mock_fetch_weather:
        response = client.get("/weather/atlanta")
        assert response.status_code == 200
        assert response.json() == {'city': 'atlanta', 'weather': 'sunny'}
        mock_fetch_weather.assert_called_once_with('atlanta')
```

### Creating a fixture that patches the function

Let's suggest that we may want to test multiple endpoints that call the `fetch_weather` function. We could patch the
function in each test, but that's a lot of code duplication. Instead, we can create a fixture that patches the
function. First, we'll set up our fixture (this is another factor function). We're using `unittest`'s Mock and Patch 
functions in order to replace the `fetch_weather` function with a mock function that returns 'rainy'. 

Note: As before, we're using a factory function to create our mock function, so we need to pass in the function itself, 
not the return value of the function.

```python
# File: tests/conftest.py
@pytest.fixture
def mock_fetch_weather_factory():
    def mock_fetch_weather(city):
        return "rainy"

    mock = Mock(side_effect=mock_fetch_weather)
    with patch.object(app.main, 'fetch_weather', new=mock):
        yield
```

In our test, we just need to again pass in the fixture as an argument.

```python
# File: tests/test_weather_api.py
def test_get_weather1(client, mock_fetch_weather_factory):
    response = client.get("/weather/atlanta")
    assert response.status_code == 200
    assert response.json() == {'city': 'atlanta', 'weather': 'rainy'}
```

### Creating a parametrized fixture

The above example probably isn't very useful in practice. In the real world, if we're testing multiple endpoints that
call the same function, we probably want to test different scenarios. For example, we may want to test that our
endpoint returns the correct response for different weather conditions. We could do this by creating multiple fixtures
that patch the function with different mock functions, but this is a lot of code duplication and basically defeats the
purpose of using a fixture at all. Instead, we can use a parametrized fixture to pass in values to our fixture to
make it behave differently depending on our test.

To do this, we're going to create another fixture, but this one will take a parameter. It essentially wraps our previous
factory function, but now we can pass in a parameter to the fixture.

```python
# File: tests/conftest.py
@pytest.fixture
def mock_fetch_weather_parametrized(request):
    def mock_fetch_weather_factory(weather):
        def mock_fetch_weather(*args, **kwargs):
            return weather

        return mock_fetch_weather

    mock_weather = request.param
    with patch.object(app.main, 'fetch_weather', new=mock_fetch_weather_factory(mock_weather)):
        yield
```

And then, when we call our test, we need to decorate it with `pytest.mark.parametrize`, and pass in the
fixture as an argument. I've set up two tests here so we can really see how it works.

```python
# File: tests/test_weather_api.py
@pytest.mark.parametrize('mock_fetch_weather_parametrized', ['cloudy'], indirect=True)
def test_get_weather_parameterized(client, mock_fetch_weather_parametrized):
    response = client.get("/weather/atlanta")
    assert response.status_code == 200
    assert response.json() == {'city': 'atlanta', 'weather': 'cloudy'}

@pytest.mark.parametrize('mock_fetch_weather_parametrized', ['tornados'], indirect=True)
def test_get_weather_parameterized(client, mock_fetch_weather_parametrized):
    response = client.get("/weather/atlanta")
    assert response.status_code == 200
    assert response.json() == {'city': 'atlanta', 'weather': 'tornados'}
```
The parameterized fixture is more complicated to set up, but it is incredibly useful in practice when you have to 
simulate different responses from external APIs.

## Mocking MongoDB

Our FastAPI application interacts with MongoDB. During testing, we might not want to (or really be able to in some cases)
hit a real database with real/mocked data.  Instead, we acn mock MongoDB by using the [mongomock](https://github.com/mongomock/mongomock) library, which 
simulates a MongoDB client. (Note: In my experience, mongomock can be difficult to work with and some practice to get 
working, but were going to proceed with it for now)

### Creating a Mock MongoDB Client

In our application, we employ context management for database interactions, utilizing Python's `with` statement. This
demands that the object used within the `with` statement must implement context management protocols, specifically
`__enter__` and `__exit__` methods.

```python
# File: app/db.py
@contextmanager
def get_mongodb():
    try:
        with MongoClient() as client:
            db = client["preferences"]
            yield db
    except Exception as e:
        print(f'Error: {e}')
        raise
```

The mongomock library doesn't implement these methods, so to align with these requirements, we define a custom 
MockMongoClient class to wrap our mongomock client. This class mimics the behavior of the actual
MongoClient by implementing the `__enter__` and `__exit__` methods. This ensures compatibility with the existing code
that expects a context-managed database client.

```python
# File: tests/conftest.py
class MockMongoClient:
    def __init__(self, db):
        self.db = db

    def __enter__(self):
        return self.db

    def __exit__(self, *args):
        pass
```

Using our MockMongoClient class, we can now create our mock database client. We'll do this in a fixture so we can
reuse it in multiple tests.

### Creating a Mock MongoDB Client Fixtures

I'm going to initialize two separate fixtures here, one with an empty database and one with some data initialized. (In
theory, we should be able to set the scope of the fixture to `session` or `module` and just initialize the database 
once, add data through other tests, and then use that data for testing later, but I haven't been able to get that 
working with mongomock. The project seems to have been designed more with unit tests in mind and is not a complete
implementation or drop in replacement. If you know how to make this work, please let me know!)

```python
# File: tests/conftest.py
@pytest.fixture
def mock_mongodb():
    def mock_get_mongodb():
        mock_client = MongoClient()
        return MockMongoClient(mock_client.db)

    return mock_get_mongodb
```

and initialize some data...

```python
# File: tests/conftest.py
@pytest.fixture
def mock_mongodb_initialized():
    def mock_get_mongodb():
        mock_client = MongoClient()
        mock_client.db.preferences.insert_one({"username": "alexjacobs", "city": "Berlin"})
        return MockMongoClient(mock_client.db)

    return mock_get_mongodb
```

And now we can write our tests. We'll pass in our fixture and again use the `dependency_overrides` feature to inject
our
mock database client into our app (overriding the `get_mongodb` function)

### Writing Tests Integration Tests Using Our Mocked MongoDB Client

We've got three simple tests here.  
First, we test that our endpoint correctly returns a 404 if no user preferences exist (we use our non-initialized
`mock_mongodb` fixture for this)

Next, we use our initialized `mock_mongodb` fixture to test that our endpoint correctly returns the user preferences.

Finally, we test that we can save user preferences (we use our non-initialized `mock_mongodb` fixture for this, but it
should work with either)

```python
# File: tests/user_preference_api.py
from app.main import app
from app.fake_db import get_mongodb

def test_get_user_preferences_404(client, mock_mongodb):
    # should return 404 since no user preferences exist
    app.dependency_overrides[get_mongodb] = mock_mongodb
    response = client.get("/users/me/preferences")
    assert response.status_code == 404
    assert response.json() == {'detail': 'Preferences not found'}

def test_get_user_preferences_initialized(client, mock_mongodb_initialized):
    # we're using our mock_mongodb_initialized fixture here which has our user preferences already set
    app.dependency_overrides[get_mongodb] = mock_mongodb_initialized
    response = client.get("/users/me/preferences")
    assert response.status_code == 200
    assert response.json() == {'city': 'Berlin'}

def test_post_user_preferences(client, mock_mongodb):
    app.dependency_overrides[get_mongodb] = mock_mongodb
    response = client.post("/users/me/preferences", json={"city": "Berlin"})
    assert response.status_code == 200
    assert response.json() == {"detail": "success"}
```

And that's it!  Pretty simple to write the actual tests once we have our mocked database client set up correctly.

## Mocking AWS S3

Our FastAPI application interacts with AWS S3 for storing and retrieving user profile pictures. During testing, we don't
want to use a real S3 bucket because it would require us to manage real AWS resources. This would complicate our tests
and potentially incur costs. Therefore, we mock the S3 bucket. (The same patterns used here could be applied to other
AWS services)

### Creating the Mock S3 Fixture

We mock the S3 bucket by using the mock_s3 decorator from the moto library, which simulates an S3 bucket. This is done
in a fixture so we can reuse it in multiple tests.

(moto is a great library for mocking AWS services, and you'll see we are able to set our fixture scope to session,
allowing us to maintain bucket state across multiple tests)

```python
# File: tests/conftest.py
@pytest.fixture(scope="session")
def mock_s3_bucket():
    with mock_s3():
        conn = boto3.resource('s3', region_name='us-east-1')
        conn.create_bucket(Bucket=bucket)
        # we could upload a test file(s) here if we wanted to
        # s3_client = boto3.client('s3', region_name='us-east-1')
        # s3_client.upload_file('tests/assets/duck.png', bucket, 'profile_pics/alexjacobs.png') 
        yield
```

### Integrating with Test Client

This looks similar to our MockMongo fixtures, but this time, rather than pass our fixture into our tests, we pass it
into the client fixture.

```python
# File: tests/conftest.py
@pytest.fixture
def client(mock_s3_bucket):
    # we patch auth within our client fixture
    from app.main import app

    def mock_get_auth():
        return TokenData(username="alexjacobs")

    app.dependency_overrides[get_auth] = mock_get_auth
    with TestClient(app) as test_client:
        yield test_client

    app.dependency_overrides.clear()
```

### Writing the Tests

Now we can write our tests. We'll start by testing that we get the right message if no profile picture exists for the
user

```python
# File: tests / test_user_preference_api.py
def test_get_user_profile_pic_404(client):
    response = client.get("/users/me/profile_pic")
    assert response.status_code == 404
    assert response.json() == {'detail': 'No profile pic found'}
```

Now we'll test adding one...

```python
# File: tests/test_user_preference_api.py
def test_set_user_profile_pic(client):
    with open('tests/assets/duck.png', 'rb') as f:
        response = client.post("/users/me/profile_pic", files={"picture": ("duck.png", f, "image/png")})

    assert response.status_code == 200
    assert response.json() == {'detail': 'success'}
```

Finally, we can test getting one. (notice how this is using the same file we uploaded in the previous test, this is due
to the fact that we're using the session scope for our mock_s3_bucket fixture)

```python
# File: tests/test_user_preference_api.py
def test_get_user_profile_pic(client):
    response = client.get("/users/me/profile_pic")
    assert response.status_code == 200
    with open('tests/assets/duck.png', 'rb') as f:
        original_image_data = f.read()
        original_base64 = base64.b64encode(original_image_data).decode('utf-8')

    assert response.json()['image'] == original_base64
```

## Wrapping Up

This was a fairly deep dive.  We've unraveled the intricacies of integration testing in FastAPI, which is not without 
its challenges when it comes to mocking external dependencies. We've gone through a
variety of techniques to mock authentication, from simple dependency_overrides to more advanced fixture-based
strategies. We've also tackled how to mock external APIs using Python's unittest.mock.patch and pytest's parametrized
fixtures.

When it comes to databases, MongoDB adds another layer of complexity. We've seen how Mongomock can be a useful tool,
albeit with its own set of limitations. We crafted custom mock MongoDB clients and fixtures to ease this pain point. As
for AWS S3, the Moto library proved to be a robust tool, enabling us to mock S3 buckets effectively, even allowing state
persistence across tests.

The aim has been to arm you with a set of tools and strategies for your FastAPI testing arsenal. Whether it's a
simple authentication mock or a more complex external service, you should now be equipped to tackle these head-on. Happy
testing.

Happy testing.

================================================================================

# CheeseGPT

- URL: https://alex-jacobs.com/posts/cheesegpt/
- Author: Alex Jacobs
- Published: 2023-03-14
- Last updated: 2026-10-12
- Tags: RAG, LLM, AI, Python, Tutorial
- Words: 2595
- Summary: A complete walkthrough of a minimal RAG pipeline in Python: scraping a corpus, chunking it, creating embeddings, storing them in Redis, and answering questions over the result with GPT.


## Introduction
This toy project was originally created for a guest lecture to a Data Science 101 course (and its quality may reflect that :)
This post extends that lecture, designed to provide a high-level understanding and example of _Retrieval Augmented Generation_ (RAG). 

We'll go through the steps of creating a RAG based LLM system, explaining what we're doing along the way, and why.
For the argument about why RAG works at all — and where it starts to break down architecturally — see
[RAG: From Context Injection to Knowledge Integration](/posts/rag/).

You can follow along with the slides and code [here](https://github.com/alexjacobs08/cheeseGPT)

![cheeseGPT](img1.png)

### The CheeseGPT System

CheeseGPT combines Large Language Models (LLMs) with the advanced capabilities
of Retrieval-Augmented Generation (RAG). At its core, CheeseGPT uses OpenAI's GPT-4 model for natural language processing.
This model serves as the backbone for generating human-like text responses. However, what sets CheeseGPT apart is its
integration with Langchain and a Redis database containing all of the information on Wikipedia relating to cheese.

When a user asks a question, the system utilizes RAG to retrieve the most relevant information/documents from its vector database, and then includes those in its message to the LLM.  This
allows the LLM to have specific and up-to-date information to use, extending from the data that it was trained on.

The image below, flow from right to left (steps 1-5) shows the high level design of this.  The user's query is passed into our embedding model.  We do a similarity search against our
database to retrieve the most relevant documents to our users question. And then these are included in context passed to our LLM.

![rag_based_llm_design](img2.png)
###### *https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1*

Below, we'll outline the steps to building this system.

**NOTE**: this is an example, and probably doesn't make a ton of sense as a useful system.  (For one, we're getting our 
data from wikipedia, which is already contained within the training data of GPT-4)  This is meant to be a high level 
example that can show how a RAG based system can work, and to show what the possibilities are when integrating external
data with LLMs  (proprietary data, industry specific technical docs, etc.)

## Data Collection and Processing

As with most projects, getting and munging your data is one of the most time consuming yet crucial elements. 
For our CheeseGPT example, this involved scraping Wikipedia for
cheese-related articles, generating embeddings, and storing them in a Redis database. Below, I'll outline these steps
with code snippets for clarity.

### Scraping Wikipedia
We start by extracting content from Wikipedia. We made a recursive function `get_page_content` to fetch pages related
to cheese, including summaries and sections.  (Note: This function could definitely be improved.)

```python

def get_page_content(page_title, depth, max_depth):
    wiki_wiki = wikipediaapi.Wikipedia('MyCheeseRAGApp/1.0 (myemail@example.com)')
    if depth > max_depth or page_title in visited_pages:
        return [], []
    visited_pages.add(page_title)

    if len(visited_pages) % 10 == 0:
        logger.info(f"Visited pages: {len(visited_pages)}")
        logger.info(f"Fetching page '{page_title}' (depth={depth})")

    try:
        page = wiki_wiki.page(page_title)
        if not page.exists():
            return [], []

        texts, metadata = [], []
        if len(texts) % 100:
            logger.info(f"Texts length {len(texts)}")

        # Add page summary
        texts.append(page.summary)
        metadata.append({'title': page_title, 'section': 'Summary', 'url': page.fullurl})

        # Add sections
        for section in page.sections:
            texts.append(section.text)
            metadata.append({'title': page_title, 'section': section.title, 'url': page.fullurl})

        # Recursive fetching for links
        if depth < max_depth:
            for link_title in page.links:
                link_texts, link_metadata = get_page_content(link_title, depth + 1, max_depth)
                if len(link_texts) > 0:
                    texts.extend(link_texts)
                    metadata.extend(link_metadata)

        return texts, metadata
    except Exception as e:
        logger.error(f"Error fetching page '{page_title}': {e}")
        return [], []

```

This is a very greedy (and lazy) approach.  We don't discriminate at all, and we end up 
with a ton of noise (things not related to cheese at all), but for our purposes of example, it works.

### Generating Embeddings
Next, we need to generate our embeddings from our collected documents. 

#### What are embeddings?

Embeddings are high-dimensional, continuous vector representations of text, words, or other types of data, 
where similar items have similar representations. They capture semantic relationships and features in a space where 
operations like distance or angle measurement can indicate similarity or dissimilarity. 

In machine learning, embeddings are used to convert categorical, symbolic, or textual data into a form that 
algorithms can process more effectively, enabling tasks like natural language processing, recommendation systems, 
and more sophisticated pattern recognition.

With our textual data collected, we'll be using OpenAI and Langchain to generate our embeddings. There are 
lots of different ways to generate embeddings (plenty packages that run locally, too), but using OpenAI API to get them
is fast and easy for us. (and also dirt cheap)

**NOTE:**
In a true production system, there would be _much_ more consideration taken around generating embeddings.  This
is arguably the most important step in a RAG based system.  We'd need to do experimentation with chunk size to see what 
gives us the best results.  We'd need to explore our vectors to make sure their working as expected, remove noise, etc.  

#### Generating embeddings using LangChain and OpenAI

Langchain makes it very easy to do create embeddings and store them in Redis without much thought,
but this step requires extreme care to generate good results in a production system

The snippet below takes our scraped wikipedia sections, generates embeddings for them using OpenAI's embeddings API, 
and stores them in Redis.  Again, LangChain abstracts away a ton of complexity and makes this _really_ easy for us.

```python
from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
rds = Redis.from_texts(
    texts,
    embeddings,
    metadatas=metadata,
    redis_url="redis://localhost:6379",
    index_name="cheese",
)
```

## Implementation of RAG

Our RAG operates by creating an embedding of the user's question and then finding the most semantically similar documents
in our database (via cosine similarity between the embedding of our user's query and the N closest documents in our database).

We then include these documents / snippets in our request to the LLM, telling it that they are the most relevant documents based 
on a similarity search.  The LLM can then use these documents as reference when generating its response.

Here's a simplified overview of the process with code snippets:

### Embedding User Queries
The user's query is converted into an embedding using the OpenAI API. This embedding represents the semantic content of
the query in a format that can be compared against the pre-computed embeddings of the database articles.

```python
from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
query_embedding = embeddings.embed_text(user_query)
```
### Retrieving Related Articles
We then use the query embedding to perform a similarity search in the Redis database. It retrieves a set number
of articles that are most semantically similar to the query.

```python
def get_related_articles(query_embedding, k=3):
    return rds.similarity_search(query_embedding, k=k)
```

### Integrating Retrieved Data into GPT-4 Prompts
The retrieved articles are formatted and integrated into the prompt for GPT-4. This allows GPT-4 to use the information
from these articles to generate a response that is not only contextually relevant but also rich in content.

```python
def create_prompt_with_articles(query, articles):
    article_summaries = [f"{article['title']}: {article['summary']}" for article in articles]
    return f"Question: {query}\n\nRelated Information:\n" + "\n".join(article_summaries)
```

### Generating the Response
Finally, the enriched prompt is fed to GPT-4, which generates a response based on both the user's query and the
additional context provided by the retrieved articles.

```python
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "system", "content": create_prompt_with_articles(user_query, related_articles)}]
)
```

Through this process, CheeseGPT effectively combines the generative power of GPT-4 with the information retrieval
capabilities of RAG, resulting in responses that are informative, accurate, and contextually rich.

## The Chat Interface

CheeseGPT's chat interface is an important component, orchestrating the interaction between the user, the
retrieval-augmented generation system, and the underlying Large Language Model (LLM). 

For the purposes of our example, we have built the bindings for the interface, but did not create a fully interactive interface.

Let's dive into the key functions that make this interaction possible.

### Connecting to Redis

```python
def rds_connect():
    rds = Redis.from_existing_index(
        embeddings,
        redis_url="redis://localhost:6379",
        index_name="cheese",
        schema="redis_schema.yaml",
    )
    return rds
```

This function establishes a connection to the Redis database, where the precomputed embeddings of cheese-related
Wikipedia pages are stored.

### Applying Filters

```python
def get_filters():
    is_not_external_link = RedisFilter.text("section") != 'External Links'
    is_not_see_also = RedisFilter.text("section") != 'See Also'
    _filter = is_not_external_link & is_not_see_also
    return _filter
```

Filters are applied to ensure that irrelevant sections like 'External Links' and 'See Also' are excluded from the search
results.

### Deduplicating Results

```python
def dedupe_results(results):
    seen = set()
    deduped_results = []
    for result in results:
        if result.page_content not in seen:
            deduped_results.append(result)
            seen.add(result.page_content)
    return deduped_results
```

This function ensures that duplicate content from the search results is removed, enhancing the quality of the final
output.  (This is necessary in our case because we were greedy / lazy when pulling our data / generating our vectors)

### Retrieving Document Results

```python
def get_results(rds, question, k=3):
    _filters = get_filters()
    results = dedupe_results(rds.similarity_search(question, k=k, filter=_filters))
    return results
```

This key function performs a similarity search in the Redis database using the user's query, filtered and deduplicated.

### Formatting RAG Results

```python
def format_rag_results(rag_results):
    divider = "*********************RESULT*********************\n"
    return [f"{divider} {result.page_content} ({result.metadata['url']}#{result.metadata['section']})" for result in
            rag_results]
```

The function formats the search results, making them readable and including the source information for transparency.

This is what our message looks like when we send it to GPT-4.  Our system prompt is first and includes instructions for the
model to use the retrieved documents when answering the question.  
 
In our user message, you can see the user's question, and then the documents we retrieved, presented as a list with some formatting.
```json {lineNos=false}
[
   {
      "role": "system",
      "content": "You are cheeseGPT, a retrieval augmented chatbot with expert knowledge of cheese. You are here to answer questions about cheese, and you should, when possible, cite your sources with the documents provided to you."
   },
   {
      "role": "user",
      "content": "User question: what is the biggest cheese sporting event.  Retrieved documents: ['*********************RESULT*********************\\n The Lucerne Cheese Festival (German: Käsefest Luzern) is a cheese festival held annually in Lucerne, Switzerland. It was established in 2001 and is normally run on a weekend in the middle of October at the Kapellplatz (Chapel Square) in the city centre. The next festival is planned to take place on 14 October 2023.The event features the biggest cheese market in central Switzerland, and offers the greatest selection of cheeses. As well as the cheese market and live demonstrations of cheesemaking, typical events during the festival include a milking competition and music such as the Swiss alphorn.The 2012 event featured over 200 varieties of cheese over 23 market stalls, including goat and sheep cheese. The 2020 event was almost cancelled because of social distancing restrictions during the COVID-19 pandemic, but was approved a few days before with a strict requirement to wear masks. Instead of the Kapellplatz, the festival was run from the nearby Kurplatz (Spa Square). 288 variety of cheeses were available at the festival, including cheesemakers from outside the local region such as the Bernese Jura and Ticino, who had their own festivals cancelled. Around 5,800 people attended the festival, lower than the previous year, with around two-thirds fewer sales. The following year\\'s event continued restrictions, where customers had to taste and buy cheese at a distance, though masks were no longer mandatory. The 2022 event featured demonstrations of the cheese making process, a chalet built of Swiss cheese, and a \"cheese chalet\" hosting cheese fondue and raclette.India Times in 2014 called it out as one of 10 world food festivals for foodies. (https://en.wikipedia.org/wiki/Lucerne_Cheese_Festival#Summary)', '*********************RESULT*********************\\n In addition to sampling and purchasing more than 4,600 cheeses in the Cheese Pavilion, visitors to the show are treated to various attractions throughout the day including cheese making demonstrations, trophy presentations and live cookery demonstrations. (https://en.wikipedia.org/wiki/International_Cheese_Awards#Show features)', '*********************RESULT*********************\\n The first annual event was held in 2000 in Oxfordshire, and was founded by Juliet Harbutt. Each year it is preceded by the British Cheese Awards, a ceremony which Harbutt created in 1994, judged by food experts and farmers, in which the best cheeses are awarded bronze, silver and gold medals.\\nAll cheeses are tasted blind, and the winners can then display their awards during the public-attended festival.  There are usually a variety of events at the festival such as seminars, masterclasses, and cheesemaking demonstrations.The event moved to Cheltenham in Gloucestershire in 2005. In 2006 the Sunday of the weekend was cancelled at great cost after the venue experienced flooding, and the decision was made to return to Oxfordshire in 2007.Early in 2008 the festival was sold to Cardiff Council; subsequently the event has been held in the grounds of Cardiff Castle in 2008, 2009, 2010, and 2011.The 2012 Great British Cheese Festival was held at Cardiff Castle on Saturday, September 22, and Sunday, September 23.\\nIn 2015, Harbutt returned to her native New Zealand. (https://en.wikipedia.org/wiki/The_Great_British_Cheese_Festival#History)']"
   }
]

```

### Generating Messages for the LLM

```python
def get_messages(question, rag_results):
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user",
         "content": f"User question: {question}.  Retrieved documents: {format_rag_results(rag_results)}"}
    ]
    return messages
```

This function prepares the input for the LLM, combining the system prompt, user question, and the retrieved documents.

The integration of these functions creates a seamless flow from the user's question to the LLM's informed response,
enabling CheeseGPT to provide expert-level insights into the world of cheese.

Putting it all together might look something like...
```python
question = "what is the biggest cheese sporting event"
results = get_results(rds, question, k=3)
messages = get_messages(question, results)
print(json.dumps(messages, indent=4))
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=messages,
)
print(response)
```

## Outcomes

So, let's compare a question using our system vs. asking ChatGPT.  We'll use the same question above.

Using our system, we get this response:

```text
The Lucerne Cheese Festival in Switzerland and the International Cheese Awards are two of the biggest cheese sporting events. 

The Lucerne Cheese Festival, a yearly event in Lucerne, Switzerland, features the largest cheese market in central Switzerland 
and provides a vast selection of cheeses. The festival also includes live cheesemaking demonstrations, a milking competition, 
and music. In past years, it has presented over 200 varieties of cheese over 23 market stalls, including goat and sheep cheese. 
The festival attracted around 5,800 attendees in 2020, despite social distancing restrictions. The India Times recognized it 
as one of 10 world food festivals for foodies in 2014 [^1^].

On the other hand, the International Cheese Awards allow visitors 
to sample and purchase over 4,600 different cheeses in the Cheese Pavilion. The show features various attractions such as
cheese making demonstrations, trophy presentations, and live cookery demonstrations [^2^].

The Great British Cheese Festival, although not as large as the other two events, is also a notable cheese event. It 
hosts the British Cheese Awards, where the best cheeses are awarded bronze, silver, and gold medals, judged by food 
experts and farmers. All cheeses are tasted blindly, and the winners can display their awards during the festival. 
The festival typically includes seminars, masterclasses, and cheesemaking demonstrations [^3^].

In conclusion, both the Lucerne Cheese Festival and the International Cheese Awards are considered some of the biggest 
cheese sporting events worldwide.

[^1^]: (https://en.wikipedia.org/wiki/Lucerne_Cheese_Festival#Summary)
[^2^]: (https://en.wikipedia.org/wiki/International_Cheese_Awards#Show_features)
[^3^]: (https://en.wikipedia.org/wiki/The_Great_British_Cheese_Festival#History)"
```

And if we ask ChatGPT the same question...

![img3.jpg](img3.jpg)

I'm not sure which of these answer is _more_ correct, and it doesn't matter for the purposes of this example.  The point is that we were
able to retrieve and include our own information, external to the model, and make it use that information in it's response.  
It's clear how many amazing use cases there are for something like this!

Hopefully this high level toy example was able to shed some light on what a RAG based system may look like.  Checkout the 
additional resources linked before for more in-depth information. Thanks for reading!

### Related reading

- [RAG: From Context Injection to Knowledge Integration](/posts/rag/) — why this pattern works, and the architectural limits it runs into.
- [The Case Against pgvector](/posts/the-case-against-pgvector/) — this toy used an in-memory vector store; here's what changes when you put one in production.
- [A Production Framework for LLM Feature Evaluation](/posts/practicalaifeatures/) — deciding which LLM features are worth building at all.

### Additional resources
https://github.com/ray-project/llm-applications/blob/main/notebooks/rag.ipynb
https://github.com/pchunduri6/rag-demystified
https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1

================================================================================

# Effective Error Handling

- URL: https://alex-jacobs.com/posts/errorhandling/
- Author: Alex Jacobs
- Published: 2022-12-14
- Last updated: 2026-10-12
- Tags: UX, Error Handling, Frontend
- Words: 796
- Summary: A simple and effective approach to error handling for end users: say what failed, say what to do next, and give support a code to search for.



All too frequently, we, as developers, are faced with the task of handling errors. By developing effective messaging 
for our users, we can make their experience with our software much more pleasant. While the task of crafting these 
messages can seem mundane, it's a necessary step in making our software more user-friendly and intuitive.

And yet, all too frequently, end users will go out of their way to completely avoid even attempting to read the error message.  

And after a long hard day writing bug free code and solving customer problems, there is nothing less satisfying than a
story that contains a screenshot of an error message that explains exactly the action the user needs to take the resolve the mistake
they have made. 

Below we present a simple and effective approach to handling errors in client-side fleshware.

## An error message that requires interaction
If it's not yet clear, I'll take this opportunity to say that this is (mostly) a joke.

\
\
And simple as that, we reduce the number of bug reports we receive by 100%!

I'm certainly no UI designer, but with a little love, I expect this to be a core product feature in all modern softwares
going forward.

Checkout the fiddle below to make your own improvements!
https://jsfiddle.net/eg142dkp/2/

### Wondering how I made this?
First I had to learn javascript.  Then I had to learn CSS and HTML.  After all that,
I had to learn about Hugo and [Shortcodes](https://gohugo.io/content-management/shortcodes/)
 
By that point, I was a certified full-stack developer, and I was able to create the following shortcode to render the modal

```html
<style>
    .modal {
        display: none; 
        position: fixed;
        z-index: 1; 
        left: 0;
        top: 0;
        width: 100%; 
        height: 100%; 
        overflow: auto; 
        background-color: rgb(0, 0, 0); 
        background-color: rgba(0, 0, 0, 0.5); 

    }

    .modal-content {
        display: block;
        background-color: #fefefe;
        margin: 25% auto; 
        padding: 20px;
        border: 3px solid #dc3545;
        width: 60%; 
        height: auto;
        opacity: 1;
        border-radius: 25px;
        text-align: center;
    }

    .modal input {
        background-color: #ffffff;
        border: 1px solid #dc3545;
        border-radius: 5px;
        width: 50%;

    }

    .error-button {
        background-color: #dc3545;
        border: 1px solid #dc3545;
        border-radius: 5px;
        width: 40%;
        color: #ffffff;
        font-weight: bold;
        display: grid;
        margin: 0 auto;
    }

</style>
<div id="errorModal" class="modal">

    <div class="modal-content">
        <div style="color: #dc3545;" id='errorMessage'></div>
        <br>
        <p style="color: gray; opacity: 65%">
            type the error message to confirm you've seen it and disable the error modal
        </p>
        <input type="text" id="textinput" name="fname">
    </div>

</div>
<div class="error-button">
    <button type="submit" onclick="errorMessage('There is an error due to the way you did it')">
        Click here to display error
    </button>
</div>
<script>
    function errorMessage(displayErrorMessage) {
        let errorModal = document.getElementById("errorModal");
        let errorMessage = document.getElementById('errorMessage')
        errorMessage.textContent = displayErrorMessage
        errorMessage.style.fontWeight = 'bold';
        errorModal.style.display = "block";

        let inputBox = document.getElementById('textinput');

        inputBox.onkeyup = function () {
            var text = inputBox.value
            if (displayErrorMessage.startsWith(text)) {
                errorMessage.innerHTML = "<span style='color: " + 'black' + ";'>" + text + "</span>" +
                    "<span style='color: " + '#dc3545' + ";'>" +
                    displayErrorMessage.replace(text, '') + "</span>"
            }

            if (text === displayErrorMessage) {
                errorModal.style.display = "none";
                inputBox.value = '';
            }
        }
        console.log(errorMessage)
    }
</script>
```
and it goes right here in my `layouts/shortcodes` directory
```
.
└── layouts
    └── shortcodes
        └── error-modal.html
```
and then in my markdown file, I can just call it like this
```

```
\
Hugo is neat

## Update 12-27-22 -- OpenAI

It is worth updating this post to say that I gave this prompt to the openai `code-davinci-002` model

```javascript
/* create a button that says 'click to display error message'. 
when the button is clicked, a modal should pop up that has an error message and a text box. 
the modal should not have an x to close it. the only way to close the model is to type the error message into the text box */
```
and within seconds had a fully working implementation

Checkout openAI's fiddle for comparison.  Maybe we're all wasting our time

https://jsfiddle.net/Lkyxuc0v/

```javascript
var button = document.createElement('button');
button.innerHTML = 'click to display error message';
document.body.appendChild(button);
button.onclick = function () {
    var modal = document.createElement('div');
    modal.style.position = 'fixed';
    modal.style.top = '0';
    modal.style.left = '0';
    modal.style.width = '100%';
    modal.style.height = '100%';
    modal.style.backgroundColor = 'rgba(0,0,0,0.5)';
    modal.style.zIndex = '9999';
    document.body.appendChild(modal);
    var errorMessage = document.createElement('div');
    errorMessage.innerHTML = 'Error: You must type the error message into the text box to close this modal.';
    errorMessage.style.position = 'absolute';
    errorMessage.style.top = '50%';
    errorMessage.style.left = '50%';
    errorMessage.style.transform = 'translate(-50%, -50%)';
    errorMessage.style.backgroundColor = 'white';
    errorMessage.style.padding = '20px';
    errorMessage.style.border = '1px solid black';
    modal.appendChild(errorMessage);
    var textBox = document.createElement('input');
    textBox.type = 'text';
    textBox.style.position = 'absolute';
    textBox.style.top = '60%';
    textBox.style.left = '50%';
    textBox.style.transform = 'translate(-50%, -50%)';
    textBox.style.width = '200px';
    textBox.style.height = '20px';
    textBox.style.border = '1px solid black';
    modal.appendChild(textBox);
    textBox.onkeyup = function () {
        if (textBox.value === 'Error: You must type the error message into the text box to close this modal.') {
            document.body.removeChild(modal);
        }
    }
}
```



================================================================================

# Running Jupyter Lab Behind NGINX: Authentication and Securing the Proxy (Part 2)

- URL: https://alex-jacobs.com/posts/jupyterlab2/
- Author: Alex Jacobs
- Published: 2022-06-15
- Last updated: 2026-10-12
- Tags: DevOps, NGINX, Jupyter, Docker, AWS
- Words: 688
- Summary: Part 2 of the Jupyter Lab + NGINX reverse proxy setup: configuring Jupyter authentication behind NGINX, handling token-based auth bypass, and locking down the proxy on EC2.


#### If you haven't read [part 1](/posts/jupyterlab1/), you may want to start there.

In the [last post](/posts/jupyterlab1/), we left off with a working reverse proxy, but we couldn't access Jupyter lab due to
its auth enforcement. Because of how we're setting this up, we will be handling
authentication upstream of Jupyter Lab, and we don't want to rely on them for handling authentication. What we are going to do here is generally considered "unsafe."  
Again, if you're looking to do this for your team, check out [Jupyter Hub](https://jupyter.org/hub)--it probably makes more sense
for your use case.

To disable token auth, we will update our Jupyter Lab config.

There is an extensive config file for Jupyter Lab. In a production environment, I recommend using it (you can generate a sample file
by running `jupyter notebook --generate-config`). But, for this toy example, we will pass
our config as cmd line args. To disable token auth and to allow same-origin requests, we're going to update our Jupyter Lab
Dockerfile Entrypoint to include these arguments
```bash {lineNos=false}
"--ServerApp.token=", "--ServerApp.password=", "--ServerApp.allow_origin", "*"
```

Our Dockerfile should now look like
```dockerfile {lineNos=false}
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
      python3-pip \
      python3-dev && \
      python3 -m pip install jupyterlab

RUN useradd -ms /bin/bash jupyter
EXPOSE 8888
ENTRYPOINT ["jupyter", "lab", "--ip=0.0.0.0", "--port", "8888", "--allow-root", "--ServerApp.token=", "--ServerApp.password=", "--ServerApp.allow_origin", "*"]
```

And if we rebuild and start our docker compose again
```bash {lineNos=false}
docker compose build && docker compose up
```

We now get through to Juypter!
![jupyter lab](img5.png)

But if we try and open the Python kernel, we'll notice it's having trouble connecting.
![jupyter lab_cant_connect](img3.png)

Opening our browser dev tools shows that there is an issue with how our proxy is handling WebSockets
![jupyter lab_websockets](img4.png)

We'll have to update our Nginx config to address this.  
We will add these lines to set headers properly for WebSockets to our / location in the server block.
```nginx {lineNos=false}
...
  server {
    listen       8000;
    server_name  localhost;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_hide_header "X-Frame-Options";
        proxy_pass http://upstream_jupyter;
        
        # websocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade "websocket";
        proxy_set_header Connection "Upgrade";
        proxy_read_timeout 86400;
    }
  }
...
```
And now, if we restart our containers using the updated config, we'll see our kernel connects!

![jupyter lab_websockets](img6.png)

If you're wondering how we will handle security when we're basically giving whoever is using this a terminal
into our cloud, the answer is using AWS to isolate the instance via IAM roles/ policy. We aren't going to get too much into
that in this post, but it is a valid concern. There isn't much we can do to prevent a privilege escalation/container escape
from a sophisticated user, but we can at least not give root access.

We're going to update our Jupyter Dockerfile to have a new user, 'jupyter', and we'll run Jupyter Lab as this user.
```dockerfile {lineNos=false}
...
RUN useradd -ms /bin/bash jupyter
USER jupyter
...
```
We're also going to update our ENTRYPOINT, so the Jupyter Lab root directory is set to the Jupyter user's home directory
```bash {lineNos=false}
"--ServerApp.root_dir", "/home/jupyter", "--ServerApp.notebook_dir", "/home/jupyter"
```

Our Dockerfile should now look like

```dockerfile {lineNos=false}
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
      python3-pip \
      python3-dev && \
      python3 -m pip install jupyterlab

# add user and switch to them
RUN useradd -ms /bin/bash jupyter
USER jupyter

EXPOSE 8888
ENTRYPOINT ["jupyter", "lab", "--ip=0.0.0.0", "--port", "8888", "--ServerApp.token=", "--ServerApp.password=", "--ServerApp.allow_origin", "*", "--ServerApp.root_dir", "/home/jupyter", "--ServerApp.notebook_dir", "/home/jupyter"]
```

If we reload our site, we'll see that the working directory is now set to `/home/jupyter`, and if we try to
write to `/`, we'll get a permissions error. It's important to note that while this makes it a little more
difficult for a malicious user to take over this 'instance', we will be giving them access to the internet,
the ability to download and install packages, execute code, etc. It would not be too difficult for someone with mal
intent to get around this. Changing the user and working directory does more to help an innocent user from accidentally breaking
something.

Great! Now we have disabled token authentication, added a system user (who is now running Jupyter), and changed our notebook
directory to our user's directory! In the next post, we'll set up a task definition and deploy to ECS.


================================================================================

# Running Jupyter Lab Behind NGINX: Reverse Proxy Setup with Docker (Part 1)

- URL: https://alex-jacobs.com/posts/jupyterlab1/
- Author: Alex Jacobs
- Published: 2022-05-08
- Last updated: 2026-10-12
- Tags: DevOps, NGINX, Jupyter, Docker, AWS
- Words: 591
- Summary: Jupyter Lab and NGINX aren't alternatives — NGINX sits in front of Jupyter as a reverse proxy. Part 1 covers containerizing Jupyter Lab and wiring it to NGINX with a sidecar pattern on EC2.


## Background

Jupyter Lab is an open source web-based IDE for notebooks with Python and R support, geared towards the data science crowd.
It's a powerful, mature application with a potentially complex configuration. Our requirement was to deliver Jupyter Lab to users
so that each user would have their own isolated "instance ." There is an off-the-shelf solution for this called Jupyter Hub that probably makes
the most sense for your organization. This example will be a proof of concept on how you could roll your solution.

##

## Step 1--Jupyter Lab Docker

##### If you aren't familiar with Docker, we're going to be using it a lot here, so check out some guides

Our first step will be getting Jupyter Lab up and running in a container.
There are many Docker images available on (Docker hub)[https://hub.docker.com/] for Jupyter Lab, but since we're rolling
everything ourselves, we might as well make our own image. It also gives us more control over our code--it's also a pretty simple Dockerfile.

```dockerfile {lineNos=false}
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
      python3-pip \
      python3-dev \
    && python3 -m pip install jupyterlab

EXPOSE 8888
ENTRYPOINT ["jupyter", "lab", "--ip=0.0.0.0", "--port", "8888", "--allow-root"]
```

There are probably some good arguments for why you should use alpine or something else as the base image here, but I'm
a sucker for ubuntu. Since this isn't a Docker tutorial, I'm not going to go into great detail here about what each
line in this Dockerfile does, but assume that it installs Jupyter Lab and configures it to run at port 8888.
We'll expand on the Jupyter Lab config (and make some changes) later, but for now, this works fine.

We're going to use Docker compose to run this. Our compose file looks like
```yaml {lineNos=false}
version: "3"
services:
  jupyter:
    build:
      context: .
      dockerfile: Dockerfile
    image: jupyter
    container_name: jupyter
    ports:
      - 8888:8888
```
We can run this with `docker compose up`

This will start our Jupyter Lab container and make it available at [http://127.0.0.1:8888/lab/](http://127.0.0.1:8888/lab/)

![jupyter lab](img1.png)

### Awesome! We're part of the way there!

Next, we need to put together an Nginx docker file.

While we could use the official Nginx image, in keeping with the theme, we're going to create our own Nginx image (and it's also
_really_ simple)

```dockerfile {lineNos=false}
FROM ubuntu:20.04

RUN apt-get update && apt-get -y install nginx
COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 8000
ENTRYPOINT ["/usr/sbin/nginx"]
```

Pretty straightforward. Our config file is also pretty simple. We're going to use port 8000, and we're going to simply
forward all requests directly to Jupyter lab.

```nginx {lineNos=false}
daemon off;
error_log /dev/stdout info;

events {}
http {

  access_log /dev/stdout;

  upstream upstream_jupyter {
    server jupyter:8888;
    keepalive 32;
  }

  server {
    listen       8000;
    server_name  localhost;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_hide_header "X-Frame-Options";
        proxy_pass http://upstream_jupyter;
    }
  }
}

```
The final piece that will tie these together is our docker-compose file. Our docker-compose is pretty simple as well.
By using Docker compose, the networking between the containers is handled for us, and we can point the `jupyter` as the service
name in our nginx.conf

```yaml {lineNos=false}
version: "3"
services:
  jupyter:
    build:
      context: .
      dockerfile: Dockerfile
    image: jupyter
    container_name: jupyter
    ports:
      - 8888:8888
  nginx:
    build:
      context: .
      dockerfile: nginx.Dockerfile
    image: jupyter-nginx
    container_name: nginx
    ports:
      - 8000:8000
    volumes:
      - nginx.conf:/etc/nginx/nginx.conf
```
Now, let's head to [http://127.0.0.1:8000](http://127.0.0.1:8000), and...

![nginx jupyter lab](img2.png)
Awesome! We're being proxied to Jupyter Lab. But, we see a page requiring token auth. This is because Jupyter is currently
configured to enforce this. In [Part 2](/posts/jupyterlab2/), we'll deal with this and some other things regarding permissions, creating a user, and making a
task definition for deploying this configuration to ECS.



================================================================================

# Splitting SRA into FASTQ with SRAToolkit, Python, and Docker

- URL: https://alex-jacobs.com/posts/fastqsplit/
- Author: Alex Jacobs
- Published: 2022-04-03
- Last updated: 2026-10-12
- Tags: Bioinformatics, Python, Docker
- Words: 1278
- Summary: A simple example using Python and Docker to split an SRA file into Fastq

## Background
SRA (Sequence Read Archive) is a file format used by NCBI, EBI, etc., for storing genomic read data. It works with
multiple file types (BAM, HDF5, FASTQ). In our case, we're going to be focusing on FASTQ. The first step of many pipelines
is converting SRA into FASTQ, which will be our focus in this post. 

If you're working as an individual or a scientist, you probably want to 
go ahead and use SRA Toolkit to download your files. For our purposes here, though,
we're going to assume you already have your files downloaded (and are probably 
using an implement outside of SRA Toolkit for file i/o)

SRA Toolkit is pretty frustrating to use. It is not designed for programmatic use or as part of larger systems
(and the developers seem hostile to the idea that someone would even try and do this :astonished:). It wants you to 
do an interactive configuration on every install https://github.com/ncbi/sra-tools/issues/77
We get around this with a dumb hack to make it think we've gone through this process and configured it. That's what's happening in lines 25-26.
(it's kind of messy to create this config file like this, it would probably be better to make this as a file and copy it in, 
but for our purposes, I want to contain everything in a single file with no external dependencies)
 
## Dockerfile

We will be using Docker and Python for this, so our first step is to create a Dockerfile with the tools we need 
installed. Here's our Dockerfile--I've added comments to explain what I'm doing, but if you don't know _anything_ about Docker, 
this isn't a day one tutorial, so check out one of those first.

```dockerfile 
# Were using ubuntu 20.04 as our base image
FROM ubuntu:20.04
# Set DEBIAN_FRONTEND to non-interactive to avoid interactive configuration
ENV DEBIAN_FRONTEND=noninteractive
# Set SRA toolkit version
ENV SRATOOLKIT_VERSION="3.0.0"
ENV USER=alex
ENV DATA=/data
# change our working directory
WORKDIR /opt

    # udpate package list and install wget, python
RUN apt-get update && apt-get install -y \
    wget \
    python3 && ln -sf python3 /usr/bin/python && \
    # download and decompress our version of SRA Toolkit
    wget https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/${SRATOOLKIT_VERSION}/sratoolkit.${SRATOOLKIT_VERSION}-ubuntu64.tar.gz && \
    tar xvf /opt/sratoolkit.${SRATOOLKIT_VERSION}-ubuntu64.tar.gz
# add SRA toolkit binaries to our path
ENV PATH=/opt/sratoolkit.3.0.0-ubuntu64/bin:${PATH}

    # create our usser
RUN useradd -ms /bin/bash ${USER} && \
    # This creates a file tricking SRA Toolkit into thinking we've gone through the manual configuration
    mkdir /home/${USER}/.ncbi/ && \
    echo '/LIBS/GUID = "mock-uid"\nconfig/default = "true"' > /home/${USER}/.ncbi/user-settings.mkfg && \
    # create a data directory to work out of and give ownership to our user
    mkdir ${DATA} && chown ${USER} ${DATA}

# copy in our python script
COPY dump_fastq.py /usr/bin/

# set our user
USER ${USER}
WORKDIR ${DATA}

ENTRYPOINT ["python", "/usr/bin/dump_fastq.py"]
```

## Python
Our python script for this is pretty simple. We're assuming that our SRA file has been
downloaded. We're going to be running SRA Toolkit using the python subprocess module.
 
First, we need to check that our SRA is valid. We use the `vdb-validate` tool to do this. If we get a good
return code, we will test if it's paired-ended data. I'm not going to go into a ton of detail 
about paired vs. single-ended data but suffice it to say that it's more effective to use paired-end data. You can read more here https://www.illumina.com/science/technology/next-generation-sequencing/plan-experiments/paired-end-vs-single-read.html

In most cases, data from modern experiments is paired, but it's essential to know. In this case, it may seem like it's not helpful,
but in a production env we would be passing these files on to another step in a pipeline, and we need to be 
able to tell if we will have a single read file or multiple to configure the next step properly.
 
To determine this, we stand on the shoulders of those who've come before us, and use a version of a function described here
https://www.biostars.org/p/139422/.

Once we determine the data type, we'll pass our SRA to `fastq-dump` to split it. We're going to 
tell it to give us gzipped output files. If the result is successful, we're simply going to return the paths of the files
(which, in this case, we're just going to log to stdout rather than upload)
 

That's basically it. We've also added some simple error handling. I've commented the file below to explain better what 
we're doing.

```python
import sys
import logging
import argparse
from subprocess import run

logging.basicConfig(level=logging.INFO)
log = logging.getLogger()

def error(message='Unexpected Error', error=None):
    log.error(message)
    log.error(error) if error else None
    sys.exit(1)

def is_paired_sra(sra):
    try:
        result = run(['fastq-dump',
                      '-X',
                      '1',
                      '-Z',
                      '--split-spot',
                      sra], capture_output=True, text=True)
        # check for good return
        if result.returncode == 0:
            # get number of lines in stdout
            num_lines = len(result.stdout.splitlines())
            # 4 lines indicates single end fastq
            if num_lines == 4:
                return False
            # 8 lines indicates paired end
            elif num_lines == 8:
                return True
            else:
                # There are cases where an index may be included, and 12 lines would be output
                # for our purposes here, we are going to treat this as an error
                error('Unable to determine if SRA is paired ended')
        else:
            error('Unable to determine if SRA is paired ended')

    except Exception as e:
        error(error=e)

def validate(sra):
    try:
        # validate sra, return result, report error
        result = run(['vdb-validate', sra])
        return result.returncode == 0
    except Exception as e:
        error(error=e)

def split_sra(sra):
    # check if sra is paired
    paired = is_paired_sra(sra)
    # dump sra into fastq
    result = run(['fastq-dump',
                  '--split-files',
                  '--gzip',
                  '--outdir', results_dir,
                  sra])

    if result.returncode == 0:
        # in prod, we would likely upload the files here. For our purposes we are just going to report their local paths
        if paired:
            # if data is paired return both read 1 and read 2
            return f'{sra.strip(".sra")}_1.fastq.gz', f'{sra.strip(".sra")}_2.fastq.gz'
        else:
            # otherwise, only read 1
            return f'{results_dir}/{sra.strip(".sra")}_1.fastq.gz'

def main(sra):
    # check for valid sra file
    if validate(sra):
        # dump sra, report results
        result = split_sra(sra)
        log.info(result)
    else:
        error('SRA is is not valid')

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Split SRA')
    parser.add_argument('--sra', type=str, help='Path to SRA')
    parser.add_argument('-data_dir', default='/data', required=False, type=str, help='Data directory')
    args = parser.parse_args()
    results_dir = args.data_dir
    main(f'{results_dir}/{args.sra}')

```

## Testing out our code
Next, we're going to need some data. SRA files are often pretty large (sometimes hundreds of gigabytes).  
Typically, S. cerevisiae RNAseq datasets are pretty small but also fully functional, so we're going to 
be using one of those https://trace.ncbi.nlm.nih.gov/Traces/?run=SRR21712309
(To find this, I went to https://ncbi.nlm.nih.gov/sra, entered S. cerevisiae in the search bar, and selected the first one :smiling:)
 
Now that we have data, we can run this container using Docker. We will 
bind a directory on our host machine (`./data`) to our `/data` directory in our container. This folder is where our input data will live, and the Fastq files our script generates will be written.  

To run, we'll simply run,

```bash {lineNos=false}
docker run -v$PWD/data:/data fastq-dump --sra SRR21712309
```
and we'll see output logged to stdout

```bash {lineNos=false}
2022-10-12T01:21:16 vdb-validate.3.0.0 info: Database 'SRR21712309' metadata: md5 ok
2022-10-12T01:21:16 vdb-validate.3.0.0 info: Table 'SEQUENCE' metadata: md5 ok
2022-10-12T01:21:16 vdb-validate.3.0.0 info: Column 'ALTREAD': checksums ok
2022-10-12T01:21:17 vdb-validate.3.0.0 info: Column 'QUALITY': checksums ok
2022-10-12T01:21:20 vdb-validate.3.0.0 info: Column 'READ': checksums ok
2022-10-12T01:21:21 vdb-validate.3.0.0 info: Database '/data/SRR21712309' contains only unaligned reads
2022-10-12T01:21:21 vdb-validate.3.0.0 info: Database 'SRR21712309' is consistent
Read 8631759 spots for /data/SRR21712309
Written 8631759 spots for /data/SRR21712309
INFO:root:('/data/SRR21712309_1.fastq.gz', '/data/SRR21712309_2.fastq.gz')
```

We'll also see these files appear in our `./data` directory. The path to these will match
our file paths logged at the end

```bash {lineNos=false}
(venv) alexjacobs@Alexs-MacBook-Pro ~/r/b/fastq-dump (master)> ls -lh data
total 1354280
-rw-r--r--@ 1 alexjacobs  staff   213M Sep 27 07:30 SRR21712309
-rw-r--r--  1 alexjacobs  staff   217M Oct 11 22:57 SRR21712309_1.fastq.gz
-rw-r--r--  1 alexjacobs  staff   221M Oct 11 22:57 SRR21712309_2.fastq.gz
```

And that's it!  This is a lot of explanation for a simple toy example, but hopefully is helpful
to someone just getting started!