AI log, day 05: evals, or how you tell better from felt-better

Day 04 ended on the ordering of fixes: prompt before retrieval, retrieval before fine-tuning. But every one of those fixes has the same hidden dependency I skipped over, which is how you actually know a change helped. Day five is about that, and it turned out to be the least glamorous and most important thing in the whole log so far. Without evals, every improvement is a vibe. You tweak the prompt, you run your one favorite test input, the output looks nicer, you ship it, and you have no idea whether you just made the other ninety cases worse.

The trap I kept falling into

The default way to test a prompt change is to run it once, read the output, and decide it’s better. I did this constantly before I forced myself to stop. The problem isn’t that it’s lazy, it’s that it’s actively misleading. You picked the one example you were thinking about while writing the change, so of course the change helps that example. That’s not evidence, it’s the setup selecting for its own success.

The tell is when you fix one thing and a week later something unrelated is broken, and you can’t say when it broke because you were never watching it. A single hand-checked example is a spot check, not a measurement. It catches “did this obviously break,” it cannot catch “did this quietly regress across the inputs I wasn’t looking at.”

The smallest honest version

An eval doesn’t have to be a framework. The minimum viable version is a table: inputs in one column, what a good output looks like in another. Ten to twenty rows is enough to start being useful, and getting to that first table is most of the value, because it forces you to write down what “good” even means, which is the part hand-testing lets you skip.

You run every row through the old prompt and the new prompt and you compare. Now a change that helps three rows and quietly wrecks four is visible, where before it was invisible. The whole point is turning “it feels better” into a number you can watch move, and the number being crude is completely fine. A crude number you actually track beats a perfect one you never build.

The hard part is the second column, the “what good looks like” part, and it splits by how checkable the task is.

When the answer is checkable

Some outputs have a right answer you can test with plain code, no model and no judgment involved. Did it return valid JSON. Did it extract the correct order ID. Did the classifier pick the label the row says it should. Is the number within range. These are the easy, boring, and most trustworthy evals, and if any slice of your task can be phrased this way, that slice should be, because a plain assertion never has an opinion and never drifts.

I underrated these at first because they feel too simple to count as evaluating an AI. That’s backwards. The parts you can check with a normal test are the parts you never have to argue about, and pushing more of the task into that shape, a forced schema, a constrained output, an ID to match, is the same day 03 instinct paying off twice: a tighter interface is also a more testable one.

When the answer is a judgment call

The awkward cases are the open-ended ones. Was this summary good. Was the tone right. Was the answer helpful. There’s no assertion for those, and this is where two imperfect tools come in.

The first is a rubric a human scores against, which sounds obvious but the value is entirely in writing the rubric down. “Good summary” is not scoreable. “Names all three parties, states the outcome, under sixty words, no invented facts” is, and the act of making it concrete is what turns your gut feeling into something a second person could apply and get the same answer.

The second is using a model to grade the output, an LLM as judge. It’s genuinely useful for scale, because a human can’t score a thousand rows on every change and a model can. It’s also not free of the exact problems it’s meant to help with. A judge model has its own biases, it tends to reward longer and more confident-sounding answers, and it can be inconsistent run to run. The move that made it trustworthy for me was checking the judge against a human on a sample first. If the model and I agree on twenty rows I’ve scored by hand, I’ll trust it on the next two hundred. If we disagree a lot, the judge is measuring something other than what I care about, and I’d rather know that than average it into a dashboard.

What actually changed in how I work

The shift is small and it’s a discipline, not a tool. Before a prompt change, write down the handful of cases it’s supposed to help and a few it must not break. Run all of them, old and new, not just the one that prompted the change. Keep the table around so next week’s change gets checked against this week’s wins instead of silently undoing them.

None of that requires infrastructure. It requires deciding that “it looks better” is a hypothesis, not a result, and that the difference between the two is the whole job. The uncomfortable part is that once you have even a crude eval, some changes you were proud of turn out to do nothing, or to trade one win for two losses. That sting is the eval working. It’s the same information you had before, you just used to ship past it.

What’s next

Day 06 gets into agents and tool use: what “agent” actually means once you strip the buzzword, and why an eval habit matters even more the moment a model stops just answering and starts taking actions.

Regular posts continue in between, as always.