AI log, day 11: evaluating an agent, not just its final answer

Day 10 ended with a specific debt: day 05’s eval framing was written before chatbot, workflow, and agent had separate definitions, and an agent’s failure mode, a bad decision three turns deep inside a loop, needs a different eval shape than a single call’s right-or-wrong answer. I’ve now sat with that debt long enough to pay it down. The short version is that a single-call eval checks a point, and an agent eval has to check a path, and those are genuinely different measurement problems, not the same problem at a bigger scale.

What day 05’s evals actually checked

Go back to day 05 for a second. The smallest honest eval setup was a table: input, expected output, an assertion that checks whether the actual output matches well enough, either an exact check for closed-form answers or a rubric/LLM-judge for open-ended ones. That setup has one implicit assumption baked into it that I didn’t call out at the time, because it didn’t matter yet: the thing being evaluated is a single function from input to output. You feed it in, one model call happens (maybe with some retries or a fixed number of chained steps, the workflow shape from day 10), and something comes out the other end. Grading that is grading a point on a line.

Why that assumption breaks for an agent

An agent, in the day 10 sense, is a loop where the model decides at runtime what happens next: which tool to call, whether to call another one, when to stop. That means what the agent actually produces isn’t one output, it’s a trajectory, a sequence of decisions, tool calls, and intermediate results that eventually terminates in something you show the user. And a trajectory has failure modes a single output can’t have.

The one that matters most in practice: an agent can take a genuinely wrong turn at step three and still arrive at a correct-looking final answer at step seven, either by accident or because it self-corrected in a way that happened to work this one time. If your eval only checks the final answer, that run passes. It’ll also happily pass a run that took nine tool calls to do what a well-behaved run does in three, quietly failing the cost and latency budget from day 07 while looking identical on the output-only metric. An outcome-only eval is blind to exactly the layer where day 10 said the new failure mode lives.

There’s a second, subtler issue: for a lot of agentic tasks, there isn’t one correct trajectory. A research agent might reasonably search in a different order than another equally competent run and still land on the same correct answer. So the eval can’t just diff the trajectory against one golden path either, that would fail correct-but-different runs and defeats the purpose. What it needs to check is closer to a set of properties the trajectory should have, not a single sequence it should match.

Two layers, not one

The fix I landed on is to split the eval into two layers that check different things, run together on the same trajectory:

Outcome-level, which is exactly day 05’s original eval, unchanged. Did the final answer match the expected answer, or pass the rubric. This still matters, it’s just no longer sufficient on its own.

Trajectory-level, which is new, and checks properties of the path rather than a specific path:

# Trajectory properties, checked against the full run, not just the final output
assert tool_calls.count <= max_expected_calls        # cost/latency budget, day 07
assert no_destructive_call_without_confirmation       # tool privilege, day 09
assert not any(repeat_calls(tool_calls))              # looping/stuck detection
assert used_expected_tool_at_least_once("search_docs") # sanity: did it actually try

None of these require knowing the one true sequence of steps. They’re bounds and invariants: don’t take more steps than the task plausibly needs, don’t call a destructive tool without the confirmation gate day 09 argued for, don’t visibly get stuck retrying the same call, and at minimum touch the tool the task obviously requires. That last one catches a specific failure I hit while testing this: an agent that hallucinated a plausible-sounding answer without ever calling the lookup tool it had available, which an outcome-only eval graded as a pass because the hallucinated number happened to be close enough to right that a lenient rubric let it through.

Where an LLM-judge earns its keep here

Some trajectory properties don’t reduce to a clean assertion. “Did the agent get confused and thrash before recovering” is a real quality signal and a real thing users notice, even when the final answer is fine, a run that stumbles for four extra tool calls before self-correcting is worse than one that goes straight there, but “stumbled” isn’t a regex. This is the one place I’ll reach for an LLM-judge over the trajectory the way day 05 reached for one over open-ended text: show the judge the full sequence of tool calls and intermediate results, not just the final answer, and ask a narrow question, “did this run take an unnecessary detour,” rather than a vague “was this good.” The narrower the question, the more the judge’s verdict is worth trusting, same lesson as day 05.

The minimal honest setup, updated

Concretely, what I’d actually build for an agent, and what I think is the smallest version that’s still honest:

• A set of scripted tasks with a known-correct final answer, same as day 05.
• A max tool-call budget per task, set generously but not infinitely, tying the eval directly to the cost curve from day 07 instead of discovering it in a bill later.
• A fixed list of tool calls that should never happen without a confirmation step, enforced as a hard assertion, not a judgment call, because this is exactly the boundary day 09 argued has no clean equivalent to SQL parameterization yet, so the eval has to be the enforcement mechanism until something better exists.
• One narrow LLM-judge question over the full trajectory, reserved for the “did it wander” quality signal that assertions can’t catch.

That’s four checks, not a research paper’s worth of eval infrastructure, and it directly maps to the failure modes day 10 actually predicted: cost blowup from unnecessary loop length, tool misuse from runtime-decided calls nobody reviewed in advance, and the genuinely new failure of a loop going wrong in a way no one wrote down.

What’s next

Same as day 10, no fixed day 12 topic yet. The evals here assumed a single agent making its own decisions in isolation; the natural next gap is what changes when more than one model is involved, whether that’s a multi-agent setup or just a single agent whose tool calls hit another model behind an API, since a lot of the cost and trust assumptions from these first eleven days were implicitly about one model call at a time.