AI log, day 15: giving an agent long-term memory is harder than bolting on a database

Day 14 was about memory inside one long conversation, a running session’s context window has nowhere else to keep state, so summarizing old turns and retrieving relevant ones both trade one failure mode for another. What that entry left out is a related but different problem: memory that is supposed to survive after the conversation actually ends. That is the feature usually marketed as “the assistant remembers you,” where something said last week shows up again this week without last week’s transcript sitting in this session’s context at all. It sounds like a smaller version of day 14’s problem. In practice it is a different problem wearing the same clothes.

The obvious version is just RAG pointed at yourself

The naive build is exactly what day 01 already described for retrieval over documents: embed every fact, store it, pull back whatever looks similar to the current query. Point that same machinery at a user’s own history instead of a document set and you have a working demo in an afternoon. It also breaks in ways a document index never has to deal with, because a document index is not full of statements about itself that quietly stop being true.

Facts expire and nothing marks the expiry

A person’s job, city, or preference from six months ago and their job, city, or preference today are both sitting in the store as equally valid memories, because nothing about “store an embedding” encodes that one of them superseded the other. Retrieval at query time has no way to know which one is current, since similarity search ranks by how closely a memory matches the question, not by which one is still true. Without an explicit newer-overrides-older rule, the system can just as easily hand back the stale fact as the current one, and it will say so with exactly the same confidence either way.

Restating the same thing keeps looking like new information

People repeat themselves across sessions in slightly different words, and each restatement gets stored as its own memory. A preference mentioned five times across five months becomes five near-duplicate entries, all scoring similarly at retrieval, which dilutes the ranking instead of reinforcing the fact. A document corpus does not have this problem in the same way, because nobody keeps re-uploading a mildly reworded copy of the same PDF every week.

Importance is not the same signal as similarity

An embedding captures how semantically close two pieces of text are, not how much either one matters. “I prefer dark mode” and “my father passed away last year” can end up stored and retrieved through the identical mechanism, weighted only by how well either happens to match the current query. Nothing in a plain similarity search distinguishes an offhand comment from something that should shape every future interaction, which means the store either keeps everything at equal weight or needs a second, separate judgment about what is even worth writing down in the first place.

What actually helps, and it starts at write time

The fixes are less about smarter retrieval and more about being careful at the point a memory gets created at all. Timestamps plus an explicit supersession rule (a new statement about the same fact replaces the old one, rather than sitting alongside it) fixes the expiry problem. Deduplication before storage, checking whether something close enough already exists rather than writing every restatement as new, fixes the redundancy problem. And a separate judgment about what is worth persisting at all, distinct from what is expected to be recalled later, is the only real answer to the importance problem, because similarity search was never going to supply that signal on its own.

The throughline from day 14 holds here too: a bad write to memory is not a one-time mistake, it is noise every future session pays for, the same shape as day 07’s argument that a long agent loop compounds a cost that looked small on turn one. Reading carelessly from memory is a wrong answer today. Writing carelessly to memory is a wrong answer on every day after this one.

What’s next

No fixed day 16 topic yet. Memory turned out to split into two genuinely separate problems, within a session and across sessions, and I would not be surprised if the next entry finds a third shape of the same “where does state actually live” question.