Request timeouts: the setting everyone copies from a tutorial

Every HTTP client library ships with a timeout setting, and every tutorial for that library sets it to some round number, 30 seconds is a common one, without ever explaining why that number and not another. I copied that number into a service years ago the same way everyone does, and it was fine, right up until it made an outage worse instead of better.

The setup

The service in question called a downstream dependency, a third-party API, with a client timeout of 30 seconds. That felt generous and safe: whatever the downstream did, we’d wait a reasonably long time before giving up, rather than failing fast on something that might just be a little slow.

What I hadn’t thought about was who was calling us. An upstream API gateway sat in front of our service with its own timeout, set independently by whoever configured that gateway, at 10 seconds. Nobody had coordinated these two numbers, because they were owned by different services on different teams, configured at different times, each one reasonable in isolation.

What happened when the dependency actually got slow

The downstream dependency had a bad day and started responding in 12 to 15 seconds instead of its usual few hundred milliseconds. Our service was still faithfully waiting, well within its 30-second budget, for a response that was going to arrive eventually. But the gateway in front of us hit its 10-second timeout first, decided our service had failed, and retried the same request.

That retry landed on us as a brand new call, which itself dispatched a brand new call to the already-struggling downstream dependency, while the first call was still in flight and still eventually going to succeed. Now there were two outstanding requests to a dependency that was already slow, doing the work of one logical request twice, adding load exactly when the dependency had the least capacity to absorb it. Multiply that by every client of our service doing the same thing, and a dependency that was merely slow got pushed the rest of the way into actually falling over.

The fix: timeouts have to shrink as you go up the call chain

The real problem wasn’t that either number was wrong on its own, it’s that the two numbers had no relationship to each other. The fix is to treat timeouts as a property of the whole chain, not a per-service constant: each hop’s timeout needs to be shorter than the timeout of whoever’s calling it, with some headroom for the hop’s own processing time.

gateway timeout:       10s
  our service's downstream call timeout:  should be < 10s, with headroom
                                            e.g. 7s, not 30s

Where possible, the better version of this isn’t even a hardcoded number per service, it’s a deadline passed along with the request itself, most languages have some form of this (Go’s context.Context with a deadline is the version I use daily), so a downstream call automatically inherits whatever time budget is actually left from the original caller, instead of every hop picking its own number independently and hoping they happen to nest correctly.

func handleRequest(ctx context.Context) error {
    // ctx already carries the deadline set by whoever called us
    downstreamCtx, cancel := context.WithTimeout(ctx, remainingBudget(ctx, 2*time.Second))
    defer cancel()
    return callDownstream(downstreamCtx)
}

remainingBudget here just means: take whatever’s left on the incoming deadline, subtract a small buffer for our own processing, and use that as the downstream call’s timeout, rather than a flat constant that ignores how much time is actually left.

Why the naive version is worse than no timeout at all

An uncoordinated timeout doesn’t just fail to help, it actively makes an outage worse, because it adds retried load at precisely the moment the system has the least spare capacity to handle it. A slow dependency getting called twice as often because a caller upstream gave up early isn’t a hypothetical, it’s the exact mechanism that turns a minor degradation into a full outage, and it’s invisible until the dependency actually gets slow enough to trigger it, which is usually the worst possible time to discover it.

What actually matters

A timeout isn’t a knob you tune once per service and forget. It’s a number that only makes sense in relation to every other timeout above and below it in the call chain, and the moment those aren’t coordinated, retries start compounding the exact slowness they were meant to protect against. Setting one number in isolation, no matter how carefully chosen, doesn’t fix that, coordinating the whole chain does.