Writing idempotent API endpoints (and why it matters for retries)

A support ticket came in a while back: a customer had been charged twice for the same order. Nothing was wrong with the payment provider, and nothing was wrong with our code in the sense of a bug you could point at. What happened was simpler and more annoying, the client’s HTTP request to POST /orders timed out on a flaky mobile connection, the app retried automatically, and the second request went through and charged the card again. The first request had actually succeeded too, it just never got the response back in time.

This is the part that’s easy to forget when you’re building an API: you don’t get to decide whether clients retry. Mobile networks drop, load balancers time out and retry the next node in the pool, HTTP client libraries retry on connection resets by default, and users just tap “submit” again when nothing seems to happen. Retries are going to hit your endpoint whether you designed for them or not. The only question is whether a retry is safe.

HTTP already has an opinion on this. GET, PUT, and DELETE are supposed to be idempotent by definition, doing them twice should leave the system in the same state as doing them once. PUT /users/42 with the same body twice just sets the same fields twice, no harm done. POST has no such guarantee. POST /orders or POST /charges means “do this action,” and doing it twice means doing it twice, two orders, two charges. That’s the gap that bit us.

The fix is the idempotency key pattern, and it’s used by pretty much every payment API you’ve ever integrated with for exactly this reason. The client generates a unique key, usually a UUID, once per logical action, and sends it as a header on the request. If the request needs to be retried, the client sends the exact same key again. The server’s job is to remember which keys it has already handled and what the response was, so a retry gets the original response instead of running the handler a second time.

Here’s roughly what that looks like on the server side in Go:

func chargeHandler(store IdempotencyStore) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		key := r.Header.Get("Idempotency-Key")
		if key == "" {
			http.Error(w, "missing Idempotency-Key header", http.StatusBadRequest)
			return
		}

		if cached, ok := store.Get(key); ok {
			w.WriteHeader(cached.StatusCode)
			w.Write(cached.Body)
			return
		}

		result, err := processCharge(r)
		if err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		store.Set(key, result, 24*time.Hour)
		w.WriteHeader(result.StatusCode)
		w.Write(result.Body)
	}
}

The lookup and the store both matter here. Whatever backs IdempotencyStore, usually Redis or a database table, needs to be checked before the actual work runs, and the result needs to be saved before the handler returns, ideally as close to a single atomic operation as you can manage, since a race between two identical retries arriving at nearly the same time is exactly the scenario this pattern exists for. A SELECT ... FOR UPDATE or a Redis SETNX on the key is usually enough to make sure two concurrent requests with the same key don’t both slip through and process the charge twice.

One detail that trips people up: idempotency keys shouldn’t live forever. A day or a few days is typical. Keeping them around indefinitely means an unbounded store, and in practice a client is never going to retry a request from three weeks ago, it will have long since given up and shown the user an error. A short TTL keeps the store small and keeps the semantics close to what people actually mean by “retry,” which is “try again soon,” not “try again eventually.”

It’s worth being honest that this doesn’t come for free. It adds a header the client has to generate correctly and a stateful store the server has to maintain, which is more moving parts than a plain handler. But the alternative is a customer support queue with duplicate-charge tickets in it, and that trade only ever looks obvious after the first incident. If your API has any endpoint that isn’t naturally idempotent, meaning basically any POST that changes money, inventory, or anything else that shouldn’t happen twice, it’s worth adding the key before you need it, not after someone asks why they got charged twice for one order.