AI log, day 08: hallucination is a sampling problem, not a lying problem

Day 07 ended with a promise to come back to hallucination, past the one paragraph day 01 spent on it: the model is doing next-token likelihood prediction, not fact lookup, so a fluent wrong answer costs it nothing extra to produce compared to a correct one. That line is true, but it’s not an explanation a teammate could act on if our RAG-backed assistant confidently cited a document that doesn’t exist. This entry is the version I’d actually want in that conversation.

There is no “I don’t know” token

The assumption I had to drop first: a model isn’t choosing between “answer” and “admit I don’t know” the way a person would. At every step it’s producing a probability distribution over the entire vocabulary, and it samples from that distribution the same way whether the tokens that follow turn out true or false. Nothing in the mechanism flags “this path is unsupported by anything I actually know.” Fluency and correctness are computed by completely different processes in my head, but the model only has one process, and fluency is what it was trained to optimize.

That’s why a hallucinated citation reads exactly like a real one: same confident register, same plausible formatting, same absence of hedging. The model isn’t performing confidence as a trick. Confident phrasing is just what the most likely continuation of “here’s a citation” looks like in the text it trained on, whether or not a real source backs it.

Two different failures that feel identical from the outside

Watching an assistant hallucinate, I used to lump every case into one bucket. There are actually at least two distinct failures, and they need different fixes.

The first is the model genuinely not having the fact. Ask about something obscure, post-training-cutoff, or simply rare in its training data, and it’s extrapolating from nearby patterns rather than retrieving anything. This is the case retrieval was built for: hand the model the actual text at query time instead of hoping it memorized the fact well enough.

The second is the model having the fact available in context and still getting it wrong, because generation is autoregressive and has no backtracking (the same point day 01 made about generation in general). Once a token commits to a wrong turn, like the wrong entity in a list of similar names, the model builds fluently on top of that mistake rather than noticing and correcting it. This is not a knowledge gap, it’s closer to a typo that compounds. More context doesn’t fix this one, and can even make it worse if the extra text gives the model more similar-looking wrong turns to take.

Diagnosing which one you’re looking at matters, because “add more retrieval” only helps the first case.

Why RAG reduces it without ever eliminating it

Day 04 already made the case for retrieval over fine-tuning, and hallucination is the sharpest version of why: grounding the model in real, retrieved text collapses a huge share of the first failure mode, because the model doesn’t have to guess at a fact that’s sitting right there in its input.

It doesn’t touch the second failure mode at all. A model can be handed the exact right paragraph and still misquote a number from it, mix up two similar-looking entities inside it, or confidently synthesize a conclusion the source text doesn’t actually support. Retrieval fixes “the model didn’t know.” It does nothing for “the model knew and still slipped.” That’s the honest ceiling on what RAG buys you, and it’s lower than the marketing around RAG usually implies.

Why “just tell it not to hallucinate” doesn’t work

The instruction I see people reach for first is adding a line to the system prompt: only state facts you’re sure of, say “I don’t know” if uncertain. It helps a little and it’s nowhere near sufficient, for the same reason day 03’s point about prompting as an interface applies here: the model has no internal signal called “certainty” that a prompt instruction can address. It can’t introspect on its own confidence the way the instruction assumes, because there’s no separate confidence value sitting alongside the token probabilities that means “this is grounded in something real” versus “this is my best guess at fluent continuation.” Asking it to self-report certainty is asking it to compute a number it was never trained to produce, so what actually shifts is often just the surface phrasing, more hedging words sprinkled into an answer that’s exactly as likely to be wrong as before.

What actually helps versus what just hides it

Grounding helps, within its limits, which is retrieval and citations pointing at real, checkable text, so a wrong answer is at least verifiable against a source rather than free-floating.

Structured uncertainty helps, meaning setups that give the model an actual abstention path with real training or scoring behind it, not just a prompt instruction, so declining to answer is a legitimate output the system rewards rather than a failure the model is discouraged from producing.

Evals, day 05’s whole point, are what turn “feels more accurate” into an actual number: a test set with known-correct answers, checked automatically, so you can say the hallucination rate went from some percentage to some lower percentage instead of trusting the one example you tried by hand.

What doesn’t help, it just hides the problem: more confident phrasing in the prompt, since it changes the words around the answer, not the process producing it. Cherry-picked demos have the same failure, a demo is one favorable draw from a distribution that still produces wrong answers on other draws, and showing the good draw doesn’t change the rate. Both make the system look more trustworthy without making it more trustworthy, which is worse than doing nothing, because it removes the visible signal that would have told you to keep checking.

What actually changed in how I think about it

Hallucination stopped being a single bug with a single fix once I split it into “doesn’t know” versus “knew and slipped,” because those need different tools and neither one is solved by asking the model nicely. The honest framing is closer to error rate than to lying: you don’t eliminate it, you measure it, you ground what you can, and you build the surrounding system to assume some rate of wrong answers will get through, the same way you’d design around any dependency that fails some percentage of the time instead of pretending it never will.

What’s next

Day 09 goes back to the trust boundary day 03 flagged and never fully closed: prompt injection, the security failure that’s specific to systems built on LLMs, where data the model reads can override instructions it was given, and why it’s a different problem from hallucination even though both come from the same lack of a hard line between instructions and content.

Regular posts continue in between, as always.