AI log, day 17: a tool schema is an API contract nothing enforces

Day 16 ended without committing to a day 17 topic, guessing the next entry was as likely to leave memory behind as to find a fourth shape of it. It left memory behind. This entry started from a smaller, dumber-looking incident: a tool definition got a field renamed for consistency with a database migration, nothing about the deploy looked wrong, and a week later a handful of account lookups had quietly returned the wrong customer.

A function signature the compiler never sees

Tool calling, the mechanism day 06 covered as what actually lets an agent do something instead of just describe it, works by handing the model a schema: a name, a description, and a list of parameters with types. The model reads that schema the same way it reads everything else, as tokens in its context, and it produces a call that tries to match the shape it was shown. Day 03 already made the case that a prompt behaves like an interface rather than a suggestion. A tool schema is the sharpest version of that claim, because it is, literally, a function signature. It has a name, typed arguments, and a contract about what a valid call looks like. The difference from an actual function signature is the part that got missed until the incident: nothing here is checked before it runs.

What a normal API break does, and what this does instead

Rename a field on a normal internal API and the break is loud on purpose. A client still calling with the old field name gets a 400, a type error, a failed deploy, something that stops the request before it does anything with a value that no longer means what the caller thinks it means. That loudness is the whole point of a contract enforced by a compiler or a schema validator sitting between the caller and the callee.

Rename a field in a tool schema and nothing stops anything. The model does not have a compiler. It has whatever pattern it can construct from the tool’s current description plus whatever it half-remembers about how a request like this usually looks, and if the new required field is one it has no real information for, it does not raise an error, it fills the field with its best guess and sends the call anyway. That is the same overconfident-fill-in-the-gap behavior day 08 described for facts in prose, just relocated to a structured argument instead of a sentence. A hallucinated fact reads as a wrong sentence. A hallucinated parameter reads as a normal, well-formed function call that happens to be pointed at the wrong customer.

The break that did not look like a break

The actual rename was mundane: a lookup tool’s customer_id parameter became account_id, matching a database table rename that had already shipped everywhere else. The tool description was updated. The JSON schema was updated. What did not get updated was every place upstream that still referenced the old name in cached few-shot examples and in an older system prompt fragment nobody remembered was still loaded for a subset of traffic. The receiving service’s argument validation accepted unknown properties instead of rejecting them, which is the default, permissive behavior for most JSON schema setups unless someone deliberately turns it off. So a call carrying both a stray customer_id the model still produced out of habit and a guessed account_id it invented to satisfy the new required field went through cleanly. No error. No failed request. Just a lookup against the wrong account, often close enough in shape (a numeric ID within a plausible range) to return a real customer instead of an obvious garbage result.

Silence was the actual bug

The rename is not really the lesson here, renames happen constantly and most of them are fine. The lesson is what day 13 already got at from a different angle: a failure that happens silently is indistinguishable from success until something downstream notices, and by the time something downstream notices, the cause is a week old and buried under everything that happened since. A strict validator, one that rejects unknown properties and missing required fields instead of quietly tolerating them, would have turned this into a visible failure at the first call. And because an agent’s tool call lives inside the loop day 06 described, that rejection did not have to be a dead end, it could have gone back to the model as the result of the call, the same way any other tool error does, giving it a chance to notice the mismatch and ask for the right value instead of guessing one. The permissive validator did not prevent an error. It just made sure nobody, human or model, ever got told one happened.

Treating a tool schema like the versioned API it is

The fix that actually stuck was not “be more careful with renames,” carefulness is not a system property. It was borrowing the same discipline this blog already covers for ordinary API versioning: prefer additive changes, add the new field alongside the old one and mark the old one deprecated in its description rather than renaming in place, and give the model a release window where both names resolve to the same lookup. When a change genuinely cannot be additive, version the tool itself the way a REST endpoint gets a /v2, get_account_v2 instead of a silently reshaped get_account, so a stale cached tool definition or an old few-shot example targets a name that simply stops existing instead of a name that now means something different. And on the receiving end, argument validation should be strict by default, reject anything that does not match the schema exactly, and feed that rejection back into the same turn instead of swallowing it. None of that makes the model better at guessing. It makes a bad guess visible at the moment it happens instead of three weeks later in a support ticket.

What’s next

No fixed topic for day 18 yet. The tool-calling thread feels like it has one more entry left in it, something closer to how much a model should be trusted to choose between multiple similar tools rather than how one tool’s shape should change over time, but that is a guess about the next step, not a commitment.

Regular posts continue in between, as always.