Circuit breakers: stopping one slow dependency from taking down everything

The incident that taught me this one didn’t start with an outage. It started with a payment provider’s API going from a normal 200ms response time to something closer to 8 seconds, still returning 200s, still working, just slow. Nothing crashed. Nothing paged on its own. And forty minutes later our own API was returning 503s to customers who had nothing to do with payments, because every worker in the pool was parked waiting on a call to a service that was technically still up.

Slow is worse than down

A dependency that’s fully down is, in a strange way, the easier failure. Connections refuse immediately, calls fail fast, and whatever error handling exists gets to run. A dependency that’s slow is the dangerous one, because every caller still thinks the call is going to succeed, so it waits. If your HTTP client’s timeout is 30 seconds and the downstream service is taking 8, every request against it holds a worker thread or a connection-pool slot for 8 seconds instead of the usual 200 milliseconds, a 40x increase in how long each request occupies a limited resource. Do that across enough concurrent requests and you run out of workers, not because of a bug in your code, but because of correct code behaving exactly as written against an input, latency, that nobody bounded.

This is why “just add a timeout” gets you partway there and then stops helping. A timeout caps how long a single request waits, which is necessary, but it doesn’t stop you from immediately sending the next request into the same overloaded dependency, and the one after that, each one also burning a worker for most of its timeout window before failing. The pool empties out at basically the same rate whether the timeout is 30 seconds or 3, it just takes longer to notice with the longer one.

What retries do to an already-struggling dependency

The instinctive fix, retry the failed call, makes this specific failure mode worse, not better. A dependency returning slow responses because it’s overloaded gets handed more load from every caller that retries on timeout, exactly when it can least afford more load. I’ve seen a retry-with-backoff policy that was perfectly reasonable for transient network blips turn a recoverable slow patch into a full outage, because every one of a few hundred callers retried two or three times each, and the downstream service that might have recovered on its own got a multiplied wave of requests instead of a chance to catch up.

# What actually happened, roughly, across a few hundred concurrent callers
1 slow request -> times out after 30s -> retried
2nd attempt -> also slow, because downstream never got a break -> times out -> retried
3rd attempt -> downstream now has 3x the concurrent load it started with

Retries are the right tool for a dependency that failed once and is fine again. They are the wrong tool for a dependency that is currently degraded, and the two look identical from a single caller’s point of view. That’s the actual problem a circuit breaker solves: it gives the system a way to tell those two cases apart, not by inspecting the downstream service, which you usually can’t do, but by watching your own failure rate against it over time.

What a circuit breaker actually does

The mechanism is simpler than the name suggests. A circuit breaker wraps calls to a dependency and tracks recent successes and failures (including slow-timeout failures, not just hard errors). Three states:

• Closed: calls go through normally. Failures are counted.
• Open: once failures cross a threshold, the breaker stops calling the dependency at all for a cooldown window, failing immediately (or falling back) instead of waiting out another timeout. This is the part that actually protects your worker pool, a call that never happens can’t hold a slot hostage.
• Half-open: after the cooldown, a small number of calls are let through as a test. If they succeed, the breaker closes and normal traffic resumes. If they still fail, it reopens and waits again.

The load-bearing insight is that “open” mode fails fast on purpose. Once you know a dependency is in trouble, the correct behavior for every subsequent request isn’t to hope this one’s different, it’s to stop asking and either fail immediately with a clear error or serve a fallback, freeing the worker in milliseconds instead of holding it for a timeout window that was never going to end well anyway.

Sizing the thresholds without guessing

The two numbers that matter are the failure-rate threshold that trips the breaker open, and the cooldown duration before it tries half-open. Both are guessable and both are wrong if you actually guess them; they come from numbers you already have.

The failure threshold should sit above your normal baseline error rate with real margin, not at some round number like 50%. If the dependency normally fails 1-2% of calls from ordinary network noise, a threshold of 10-15% over a rolling window catches genuine degradation without tripping on an ordinary bad minute. Watch it in production for a week before trusting a number here; the honest failure rate is rarely what people guess it is.

The cooldown should be sized against how long the dependency actually takes to recover from the kind of incident you’re protecting against, not an arbitrary “30 seconds feels right.” If a payment provider’s degraded periods historically last 2-5 minutes based on past incidents, a 10-second cooldown just means the breaker flaps open and closed a dozen times during the outage, doing almost no good. A cooldown in the same ballpark as your actual recovery time, checked against a monitoring dashboard rather than assumed, does the job the mechanism is meant for: not preventing the outage, but keeping your own service’s blast radius from becoming as wide as the dependency’s.

Where this actually earns its complexity

A circuit breaker is one more piece of state to reason about, so it’s worth being honest about when it earns that cost. It’s clearly worth it for any call to a dependency outside your control that the rest of your service doesn’t strictly need to function, a third-party API, an optional enrichment step, a payment provider you can queue and retry later rather than block on synchronously. It’s less obviously worth it for a call to your own primary database, where “fail fast and serve a fallback” often isn’t an option anyway, because there’s no fallback for “we don’t know the user’s data.” Reach for it where a slow dependency is separable from the request path that actually needs to keep working, not as a blanket wrapper around every outbound call in the codebase.