Health checks that lie: liveness vs readiness

We had one health check endpoint, it returned 200 if the process had started successfully, and both the orchestrator’s restart policy and the load balancer’s routing decisions pointed at it. That worked fine for a long time, because the two decisions it was answering, “should this instance be restarted” and “should this instance receive traffic,” happen to have the same answer almost always. The day our database had a ten-minute outage, they stopped agreeing, and the single check gave the wrong answer to both systems at once.

What the check actually measured

func healthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
}

That is close to what we had, slightly dressed up but not meaningfully different: a handler that answers 200 unconditionally as long as the HTTP server itself is up and able to route a request to it. During the database outage, every request that actually tried to do anything, look up a user, save a record, anything past the handler’s first line, failed. The health endpoint kept returning 200, because it never touched the database, it only proved the process could still accept a connection and run a function.

Two things happened downstream of that single wrong signal. The load balancer, seeing 200 from every instance, kept routing user traffic evenly across all of them, meaning every instance was equally busy failing every real request instead of at least concentrating the damage. And the orchestrator, also seeing 200 from every instance, never restarted anything, because from its point of view every process was healthy. Ten minutes of a fully broken user-facing service, and every automated system watching it agreed nothing was wrong.

Two different questions, wearing the same endpoint

The actual mistake was treating “is the process alive” and “can this instance serve a real request right now” as the same question, because they usually move together. A process that crashes cannot serve requests, so checking liveness and checking readiness give the same answer in the failure mode most people picture when they think “health check”: the process itself is broken. A process that is very much alive but whose one required dependency is down is a failure mode the combined check cannot see, because nothing about the process itself is wrong.

The fix: split the endpoint, split the decision

func livenessHandler(w http.ResponseWriter, r *http.Request) {
    // is the process itself stuck or deadlocked? Not "is everything working."
    w.WriteHeader(http.StatusOK)
}

func readinessHandler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
    defer cancel()
    if err := db.PingContext(ctx); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

/healthz/live answers one narrow question: is this process stuck in a way that only a restart fixes, a deadlock, an unrecovered panic loop, something a restart is the correct response to. It intentionally does not check the database, because the database being down is not a reason to kill and restart this process, restarting it would not bring the database back and would just add churn on top of an outage that has nothing to do with this instance’s own health.

/healthz/ready answers a different question: can this specific instance serve a real request right now. It checks the dependency that actually determines that, here a database ping with a short timeout so a slow database does not turn the readiness check itself into a hung request. The load balancer polls this one, and during the outage, every instance correctly reports not ready, and the load balancer stops sending it traffic instead of spreading failures evenly across a fleet that cannot do anything with them.

With the split in place, the same ten-minute database outage produces a much less dramatic incident: every instance fails its readiness check and gets pulled from rotation, the orchestrator’s liveness check keeps passing so nothing gets needlessly restarted mid-outage, and once the database comes back, readiness checks start passing again and traffic resumes without anyone touching a single process.

Why “just add a health check” undersells this

The advice to add a health check is right and also incomplete, because it implies there is one true answer to “is this instance healthy,” when there are actually at least two separate consumers of that signal asking two separate questions, and they need different answers under a dependency outage specifically, which is the one scenario where they diverge. Getting this wrong does not look like a missing feature, it looks like a system that appears to be monitoring itself correctly, passing every check, right up until the moment a dependency goes down and the whole safety net turns out to have been checking the wrong thing the entire time.