Distributed locks: the mutex that wasn't one

A nightly billing job invoices every active customer once, and it is supposed to be the only thing touching that table at a time. During a rolling deploy, the old instance was still finishing its run when the new instance started up and kicked off its own. Same job, same customers, same night, twice. A handful of people got charged twice, and the fix that went in the next morning was the standard one: a Redis lock, acquired with SET key value NX PX 30000 before the job starts, released when it finishes. One key, one holder, problem solved. Except the same double-charge happened again three weeks later, on a night with no deploy at all.

The job outlived its own lock

The billing run had grown slower as the customer list grew, and on that particular night it took longer than the 30 second TTL picked when the lock was first added. The lock expired while the job was still in the middle of its loop. A second scheduled trigger came in, saw no key, acquired a fresh lock, and started a second full run against the same table the first run was still writing to. Both processes were, from Redis’s point of view, doing exactly what they were told: check for a key, and if it is not there, you have the lock. Neither one had any way of knowing the other believed the same thing at the same time.

A TTL is a guess about how long the work takes, not a guarantee about who is doing it

The lock’s only signal for “is someone still working” was whether the key still existed, and the key existing was purely a function of a timer set before the work even started. That timer has no relationship to how long the actual job takes on any given night, a slightly larger customer batch, a slower query plan, a moment of contention on the database, and the estimate is wrong. Once it’s wrong, the lock does not fail loudly. It just quietly stops meaning what everyone assumed it meant, and a second worker walks in through a door that looks closed but isn’t holding anything shut anymore.

The release code had the same blind spot

There was a second bug hiding behind the first one: the code that released the lock at the end of the job just did DEL key, unconditionally. If the first job’s lock had already expired and a second job had acquired a new one, the first job finishing its (now pointless) work would delete the second job’s live lock on its way out, the one actually protecting real, in-progress work. A lock that anyone can release, not just the one who acquired it, is not really a lock. It is a shared light switch that whoever gets to it last gets to flip, regardless of who turned it on.

What actually fixed it

Two changes, neither of which touches the TTL number itself. First, every acquisition writes a random unique token as the lock’s value, not a fixed placeholder, and release becomes a compare-and-delete: only remove the key if its value still matches the token this worker acquired, done atomically in a small Lua script so the check and the delete cannot race each other. That alone stops a worker from ever releasing a lock it no longer holds. Second, the job renews its own lock on a heartbeat while it runs, extending the TTL every few seconds instead of betting the whole run on one number guessed in advance. A worker that dies mid-run stops heartbeating and its lock expires for real, which is the one case a TTL is actually good for. A worker that is merely slow keeps its lock exactly as long as it is genuinely still working.

The part a heartbeat still doesn’t fix

None of this closes the theoretical gap that shows up in any lock backed by a single system with its own pauses and network hiccups, a worker can still stall between its last heartbeat and the moment it writes to the table, long enough for the lock to expire and a second worker to step in before the first one’s write actually lands. That is the same critique Martin Kleppmann made of Redis-based locking in general: a TTL-based lock can promise mutual exclusion most of the time, not all of the time, unless the protected resource itself can reject a write from a holder it knows is stale. The honest fix for a job where “most of the time” isn’t good enough is a fencing token, an ever-increasing number handed out with each lock acquisition, checked and rejected by the database itself if a write arrives with a token older than one it has already seen. The billing job didn’t need that level of rigor to stop double-charging in practice, but it’s worth knowing the lock alone was never actually a hard guarantee, just a very good reduction in how often the race gets a chance to happen.