Graceful shutdown: handling SIGTERM instead of getting killed mid-request

For a long time, every deploy on our single VPS setup had a small, consistent cost: a handful of requests failed, every single time, without fail. Not many, maybe three or four out of a few thousand, but they were real users getting a real error, on a schedule, every time we shipped. It took embarrassingly long to admit that “a few requests fail on every deploy” was not an acceptable steady state, it had just been happening long enough that it stopped registering as a bug.

What was actually happening

The deploy script sent SIGTERM to the running process and gave it a couple of seconds before following up with SIGKILL if it hadn’t exited. The Go binary had no signal handling of its own, so the default behavior kicked in: SIGTERM terminates the process immediately. Any request that happened to be mid-flight when that signal arrived, a database query half-executed, a response half-written, just stopped existing. The client got a connection reset, not a clean error, because there was no clean anything, the process was gone.

This is invisible in testing because nobody deploys during a load test. It only shows up under real traffic, mid-day, whenever whoever is shipping that day happens to hit deploy, and it looks like a rare flaky failure rather than a deterministic one, because the request that gets caught in it is different every time.

The fix: catch the signal, drain, then exit

The actual fix in Go is a well-known pattern once you go looking for it, but it doesn’t happen by default, you have to opt in:

func main() {
    srv := &http.Server{Addr: ":8080", Handler: router}

    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("server error: %v", err)
        }
    }()

    stop := make(chan os.Signal, 1)
    signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
    <-stop

    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("forced shutdown after grace period: %v", err)
    }
}

srv.Shutdown stops accepting new connections immediately, but lets any request that’s already in flight finish normally, up to the 15-second deadline on the context. A request that was mid-flight when the signal arrived now gets to complete and return a real response instead of getting cut off mid-write. Only if something is still running past the grace period does it get forced closed, which is a deliberate last resort, not the default path.

Fifteen seconds was picked by looking at our actual p99 request duration and giving it several times that as headroom, not a number copied from an example. A grace period shorter than your slowest realistic request just recreates the original bug for that one request, so it’s worth checking your own latency numbers before picking one.

The part that’s easy to miss: the load balancer doesn’t know yet

Fixing the process’s own shutdown wasn’t quite enough on its own. There’s a gap between “this instance stopped accepting new connections” and “the load balancer or reverse proxy has noticed and stopped sending it new ones,” because that decision usually depends on a health check that runs every few seconds, not instantly. During that gap, new connections can still arrive at an instance that’s already draining, and immediately get a connection refused instead of being routed somewhere healthy.

The fix there was to flip the instance’s own health check to unhealthy the moment the shutdown signal is received, before starting the drain, and give the proxy a beat to notice and pull it out of rotation before the grace period timer really starts mattering. It’s a small ordering change: mark unhealthy, wait briefly, then start draining, rather than doing both at once.

Why this gets skipped

None of this is complicated once it’s written down, which is exactly why it’s easy to skip. It costs nothing and changes nothing about how the service behaves 99.9% of the time. The only moment it matters is a deploy, which is also the one moment a developer is watching a terminal instead of watching the requests actually happening in production, so the failed requests never really get seen by the person shipping the code that day. Nobody standing in front of a “deploy successful” message is also tailing the error rate for the ten seconds after.

The instinct to skip it is reasonable, right up until you count how many deploys happen over a year and multiply by the handful of requests each one drops. A clean shutdown is maybe twenty lines of code that only pays off during the one operation nobody is watching closely, which is precisely why it’s worth writing before the first deploy, not after the first complaint.