The DNS TTL nobody lowered until it was too late

I planned a server migration carefully. New box provisioned, app deployed, database replicated over, everything smoke-tested against the new server’s IP directly. The only step left was flipping the domain’s A record from the old IP to the new one, which I did, confirmed the change had propagated to the registrar’s nameservers, and moved on to decommissioning the old server a few hours later. That decommissioning is the part that went wrong, and it went wrong for a reason that had nothing to do with anything I’d actually tested.

What the TTL had actually been set to

The A record’s TTL had been sitting at whatever the DNS provider defaulted to when the domain was first set up, years earlier, which in this case was 86400 seconds, a full day. Nobody had ever revisited it, because there had never been a reason to. A TTL only matters at the moment you need to change the record, and until this migration, the record had never needed to change.

example.com.    86400    IN    A    203.0.113.10

I updated the record to point at the new IP, and the update itself went through the registrar instantly. What I hadn’t accounted for is that the update only changes what the authoritative nameserver answers going forward. Every resolver out in the world that had already asked for example.com in roughly the last day, ISP resolvers, corporate DNS caches, individual machines with their own local cache, had that old answer sitting in memory with up to 24 hours left before it would expire and ask again. I decommissioned the old server four hours after the record change, and for most of the rest of that day, a real slice of visitors were still being handed the old IP by their own resolver’s cache and hitting a server that no longer existed.

Why this doesn’t show up until the cutover

Nothing about a stale TTL is visible while you’re not migrating. The record answers correctly, requests resolve correctly, and a TTL of a day versus a TTL of five minutes produces identical behavior for every single request except the one class of request that matters exactly once: the first request from a given resolver after you’ve changed the answer. You can run months of ordinary traffic against a day-long TTL without ever noticing it’s set that way, because ordinary traffic doesn’t require the record to change out from under it. The cost is deferred entirely to migration day, which is precisely the day you have the least slack to absorb a surprise.

The fix has to happen before the migration, not during it

The actual fix isn’t a DNS setting you flip during the cutover, because by the time you need the fast propagation, the slow TTL has already been cached everywhere it’s going to be cached. It has to happen in advance:

• Days before the planned cutover, lower the TTL on the existing record, in this case from 86400 down to something like 300 seconds, while it’s still pointing at the old server.
• Wait out the old, longer TTL. Any resolver that cached the record before you lowered it is still holding onto whatever TTL it saw at the time, so you need the full old TTL window to pass (a day, in this case) before you can be confident every cache has re-fetched and picked up the new, shorter one.
• Only then make the actual IP change. With a 300-second TTL now in effect everywhere, the propagation window for the real cutover shrinks from most of a day to a few minutes.
• Once the migration is confirmed stable, raise the TTL back up if a long TTL is otherwise the right choice for that record (less load on the nameserver, marginally faster resolution for end users).

The mental model that actually sticks

A TTL isn’t a setting you use during a change, it’s a commitment you make about how fast you’re allowed to change your mind, and that commitment gets bought back over an entire prior TTL window’s length of caching, not at the moment you need it. If you know a cutover is coming, the TTL has to shrink days ahead of time, not on the day of. By the time an outage from a stale cache is happening, the TTL that caused it already expired the option of fixing it quickly.