AI log, day 14: context windows are memory, and memory is not free

Day 13 ended pointing at context, so that’s this entry. Day 07 already made the cost argument: an agent loop re-sends its growing context every single turn, which is why a long loop trends toward quadratic cost instead of linear. What I hadn’t sat with directly is the framing underneath that cost: the context window isn’t just an input format, it’s the entire memory a running agent has. There’s no separate place anything “sticks.” Whatever it needs from ten turns ago has to still physically be sitting in the context, getting re-read and re-billed on turn eleven, twelve, and every turn after that.

There is no other place for it to live

This is easy to miss because a chat interface feels like it remembers things the way a person does, a fact mentioned once, still true later without being repeated. Under the hood that’s not what’s happening. Every turn, the whole prior conversation, or as much of it as fits, gets sent again as part of the input. “Remembering” a detail from turn two on turn twenty isn’t recall, it’s re-reading, because the detail is still physically present in the text going into the model on every single turn between those two points. The context window is doing the job memory does in a normal program, holding state across steps, except it’s billed and bounded in a way ordinary memory isn’t. A variable sitting in RAM doesn’t cost anything per read. A fact sitting in a context window costs a little more, every turn, for as long as it’s in there.

That’s fine for a short conversation. It stops being fine once a session runs long enough that the growing context itself becomes the majority of what’s actually driving cost and latency, day 07’s argument, but now with the added wrinkle that the model window itself has a hard size limit. Eventually there’s more history than fits, and something has to give.

Response one: compress what’s already there

The first practical answer is summarizing older turns into a shorter form, replacing forty turns of back-and-forth with a paragraph that captures the parts that still matter, and keeping the recent turns verbatim. This buys back room in the window and keeps cost from scaling with the full session length. The risk is exactly what it sounds like: summarizing is lossy by definition, and the thing that gets dropped is, by construction, whatever the person doing the summarizing (a smaller model, usually, running its own pass over the history) judged least important at the time. The problem is that “least important at the time” and “least important to a question asked five turns later” aren’t the same judgment. A detail that looked like throwaway color in turn six can turn out to be exactly what turn thirty needed, and by then it’s gone, replaced by a paragraph that didn’t think it mattered.

Response two: retrieve only what’s relevant

The second answer, closer to day 01’s original framing of embeddings and retrieval, is to not keep everything in the window at all, store the full history somewhere external, and pull back only the pieces that look relevant to the current turn, the same retrieval idea from that entry, just pointed at an agent’s own past instead of an outside document set. This avoids the lossy-compression problem, nothing gets rewritten or thrown away, but trades it for a different one: relevance at retrieval time is a guess, made by whatever similarity search is doing the pulling, and a guess can miss. Something that didn’t look related to the current query, by whatever metric the retrieval step is using, doesn’t get pulled back, even if it was exactly the fact this turn needed. Retrieval doesn’t lose information the way summarization does. It loses access to information, which from the model’s side looks identical to it not existing.

Neither one is a solved problem

Both responses are real, both get used in practice, and neither is free in the way it’s sometimes described. Compaction risks discarding the exact detail a later turn needed, chosen by a judgment made before that need was visible. Retrieval risks failing to surface the exact detail a later turn needed, missed by a relevance judgment made before that need was visible. Different mechanism, same shape of failure: a decision about what matters, made too early, by something that can’t see the question that will eventually depend on the answer. Context management isn’t an implementation detail to configure and forget. It’s a design decision with its own honest failure modes, the same way the eval setup from day 05 and day 11 had its own honest failure modes instead of a clean, complete answer.

What’s next

No fixed day 15 topic yet. Fourteen entries in, the throughline connecting most of them is that very little about running a model in a loop is actually free, cost, trust, guardrails, now memory, and I’d rather keep finding the next hidden cost than declare the list finished.