Retries need backoff and jitter, or they become the outage

The outage I want to talk about started with something that should not have been an outage at all: a four-second blip in a downstream pricing API. Nothing failed hard, nothing crashed, it just got slow enough for a moment that a batch of calls timed out. What actually took the service down for the next six minutes was not the blip. It was every one of our own workers retrying on the same fixed schedule, all at once, over and over, in perfect lockstep.

A retry loop that looked completely reasonable

The retry code had been sitting in the codebase for a couple of years, and it looked fine on every review it ever got:

func fetchPrice(ctx context.Context, sku string) (Price, error) {
    var lastErr error
    for attempt := 0; attempt < 3; attempt++ {
        price, err := priceAPI.Get(ctx, sku)
        if err == nil {
            return price, nil
        }
        lastErr = err
        time.Sleep(500 * time.Millisecond) // retry after a fixed delay
    }
    return Price{}, lastErr
}

Three attempts, half a second apart, called from a couple hundred concurrent request handlers. On any ordinary day, that is completely harmless, because failures are rare and scattered in time. The day it mattered, the pricing API had a genuine four-second slowdown, and something close to all of those couple hundred handlers timed out on their first attempt within the same rough window. Half a second later, all of them retried at once. Half a second after that, the ones that failed again retried again, still together. The pricing API, which had mostly recovered from its original blip, was now getting hit with three synchronized waves of retry traffic stacked on top of its normal load, each one bigger than the last because handlers that failed on attempt one and attempt two piled their attempt three on top of everyone else’s attempt one. What should have resolved itself in under five seconds instead took the dependency down for six minutes, entirely from the shape of our own retry traffic.

The fix: back off, and stagger who retries when

The fix has two separate parts, and it is worth being clear that they solve two different problems. Backoff means each successive retry waits longer than the last, so a struggling dependency gets progressively more room instead of a constant hammering. Jitter means that room is randomized per caller, so hundreds of clients that failed in the same half-second window do not all wake up and retry in the same next half-second window.

func fetchPrice(ctx context.Context, sku string) (Price, error) {
    var lastErr error
    base := 200 * time.Millisecond
    for attempt := 0; attempt < 3; attempt++ {
        price, err := priceAPI.Get(ctx, sku)
        if err == nil {
            return price, nil
        }
        lastErr = err

        // exponential backoff: 200ms, 400ms, 800ms...
        backoff := base * time.Duration(1<<attempt)
        // full jitter: sleep somewhere in [0, backoff), not exactly backoff
        jittered := time.Duration(rand.Int63n(int64(backoff)))

        select {
        case <-time.After(jittered):
        case <-ctx.Done():
            return Price{}, ctx.Err()
        }
    }
    return Price{}, lastErr
}

Backoff alone would have helped a little, spreading the three attempts further apart in time. It would not have fixed the actual problem, because every caller was still backing off on the same schedule, so attempt two would still land in one synchronized wave, just a later one. Jitter is the part that actually breaks the synchronization: two callers that both failed at the same instant now retry at two different, unpredictable moments instead of the same one, so the retry traffic spreads out into something closer to the dependency’s normal request pattern instead of a series of coordinated spikes.

Why this is easy to miss

Nobody sets out to write a retry storm. The loop gets added because a single call failing occasionally is a real problem worth handling, and a fixed delay is the first thing anyone reaches for because it is the simplest correct-looking version of “wait a bit and try again.” It behaves exactly as intended in every test anyone runs, because tests fail one request at a time, never a couple hundred at once. The failure mode only exists at a scale that a single developer testing locally cannot reproduce: it requires many independent callers failing at roughly the same moment, which only really happens when the thing they all depend on has an actual problem, which is exactly the moment you can least afford your retry policy to make things worse.

The mental model that stuck

The reframe that made this obvious in hindsight: a retry without jitter is not “try again in a bit,” it is “guarantee that everyone who failed together retries together.” Backoff controls how much load a struggling dependency gets over time. Jitter controls whether that load arrives as a smooth trickle or a series of synchronized spikes shaped exactly like the traffic pattern that got you into trouble in the first place. Both numbers matter, but only one of them stops your own clients from acting like a single, larger, more precisely timed version of the same problem.