AI log, day 07: a model call is a billed API call, not free compute

Day 06 ended on the idea that an agent is a loop that can call the model several times to answer once, and I said the economics of that loop deserved their own entry. This is that entry. Coming at this as a backend engineer, I have a reflex I trust more than almost any other: before I ship an endpoint, I want to know roughly what one request costs and how long it takes. For a plain API that’s database time plus some CPU. For anything that calls a model, both numbers work differently than I expected, and the agent loop from day 06 quietly multiplies both.

Tokens are the unit of the bill and the wait

The thing that reframed this for me is that tokens, the same tokens from day 01, are the unit of two separate things at once: what you pay, and what you wait for.

On cost, you’re billed for tokens in and tokens out, and they’re priced differently, output usually costing more per token than input. So a request that sends a short prompt and gets a long answer is charged very differently from one that sends a huge document and gets back one word, even though I’d have lazily called both “one model call.”

On latency, the dominant cost is output tokens, because the model generates them one at a time, in sequence. A 500-token answer genuinely takes longer to produce than a 50-token answer, in a way that has no equivalent in a normal function return. There’s also a fixed-ish delay before the first token shows up, time to first token, and then a steady stream after that. So “make the model answer faster” mostly means “make the model answer shorter,” which is not an instinct I brought in from backend work.

The agent loop multiplies both, and not linearly

Here’s the part that actually surprised me. On day 06 the loop looked cheap because each step looked cheap. But a loop that calls the model five times to answer once isn’t just five times the cost and five times the latency. It’s worse, because the conversation grows.

Every turn, you append the tool result to the context and send the whole thing back. The model is stateless between calls, so turn three re-sends everything from turns one and two, plus the new tool output. The input on the last call of a long loop can be many times the input on the first. So the cost of a loop isn’t n times a fixed call, it creeps toward quadratic in the number of turns, because each turn re-pays for all the context before it.

turn 1: send [prompt + tools]                          -> small input
turn 2: send [prompt + tools + call1 + result1]        -> bigger
turn 3: send [prompt + tools + call1..2 + result1..2]  -> bigger still
...

That matched a vague unease I’d had watching an agent “think” for a while and then hand back a short answer. The short answer was cheap. The pile of context it re-read on every step to get there was not.

The backend levers mostly transfer

Once I saw it as a metered, billed dependency, the tools I already reach for on any expensive endpoint mostly carried over.

Cache the stable part. If the front of your context is the same every call, a long system prompt, a fixed set of instructions, prompt caching lets you avoid re-paying full price to re-send it each turn. It’s the same instinct as not recomputing something constant inside a loop.

Right-size the model. Not every call in the loop needs the biggest model. A cheap, fast model is often fine for the routine steps, with the expensive one reserved for the hard reasoning, the same way you don’t run every query against the primary if a replica will do.

Cut what you send. Trimming stale tool results out of the context, or summarizing earlier turns instead of carrying them verbatim, attacks the quadratic growth directly. Less context per turn is both cheaper and faster.

Cap the loop. Day 06 argued for an iteration cap so a confused agent fails instead of spinning. It turns out to be a cost control too. The cap is the ceiling on your worst-case bill per request, which is exactly the kind of number I want to be able to state before shipping.

And for latency specifically, streaming the answer as it generates hides the wait the same way progressive rendering does on the web. It doesn’t make the total faster, but time to first visible token is what a person actually feels.

What actually changed in how I think about it

The shift was small and it fixed a lot. I stopped treating a model call as compute I own and started treating it as an external, per-request-billed API call, like a paid third-party service sitting in my request path. The moment I put it in that mental box, all the right questions came for free: what does one call cost, how many do we make per user request, what’s the p95 latency of the whole loop, and what’s the ceiling if it goes long.

None of that is exotic AI knowledge. It’s the same cost-per-request discipline I’d apply to any endpoint that calls something I pay for and don’t control. The only genuinely new part is that the unit is tokens, the loop re-pays for its own history, and “faster” usually means “shorter.” Everything else is a backend problem wearing a new dependency.

What’s next

Day 08 goes back to hallucination, a proper mechanical dive past the one-paragraph version in day 01: what’s actually happening when a model states something false confidently, and which fixes reduce it versus which ones just hide it.

Regular posts continue in between, as always.