Testing in production sounds reckless until you realize you already do it

“Never test in production” is one of those lines that sounds obviously true right up until you look at how any of us actually ship software. I believed it uncritically for years, treated it as a settled question, and then noticed that every deploy process I’d ever trusted, including the careful ones, was quietly full of production testing that nobody called by that name.

The deploy that “wasn’t” a test, and clearly was

We had a change to how a pricing calculation rounded fractional cents, tested thoroughly in staging against a fixed set of sample orders, all green. Rolled it out. Watched the error rate dashboard for the next twenty minutes the way we did for every deploy, and about twelve minutes in, a spike showed up: an edge case where a specific discount combination produced a negative line-item total that a downstream validation rejected, something none of our staging fixtures happened to combine.

That deploy had staging tests, a code review, and still needed a live rollout followed by someone actually watching what happened, because the fixture set in staging, however careful, was a guess at the shapes of real orders, and the guess was wrong in one combination nobody had thought to construct. The twenty minutes of watching the dashboard after the deploy wasn’t a formality I did out of habit. It was the actual test that caught the actual bug. Staging didn’t catch it. Production did, because production is the only place real order combinations exist.

Every safe deploy practice is a production test wearing a different name

Once I noticed that one, I started noticing the pattern everywhere in our own process:

• A canary release that ships to 5% of traffic before the rest is, functionally, running the new code against real production requests and checking the results before committing further. That’s a test. It happens in production, on purpose, because that’s the only place the input distribution is real.
• A feature flag gated to internal accounts lets new code run against a live production environment, with production data, production latency, and production integrations, before any external user sees it. Also a test, also in production, also deliberate.
• Watching error rates and latency percentiles for the fifteen minutes after any deploy, which every team I’ve worked on does whether or not anyone calls it that, is a manual acceptance test with production traffic as the input.

None of these are accidents or lapses in discipline. They’re the actual mechanism by which software gets validated against the one environment staging can never fully replicate: real traffic shape, real data skew, real concurrent load, real third-party services behaving the way they actually behave instead of the way a staging stub assumes they behave. Staging is a hypothesis about production. Every one of these practices is a way of checking the hypothesis against the thing itself, cheaply, before betting everything on it being right.

What “never test in production” is actually trying to prevent

The rule isn’t wrong about the danger, it’s just aimed at the wrong verb. The dangerous move was never “production sees this code.” The dangerous move is shipping a change to all of production traffic with no way to detect that something went wrong and no fast way to undo it. A canary at 5% with monitoring and an automatic rollback trigger is a controlled production test. A deploy straight to 100% with no dashboard open and no rollback plan is an uncontrolled one, and it’s uncontrolled in exactly the same environment either way. The environment was never the risk. The blast radius and the recovery time were.

What actually changed for me

I stopped treating “in production” as the disqualifying fact about a practice and started asking two questions instead: how much of production is exposed to this before I know whether it worked, and how fast can I detect and undo it if it didn’t. A rounding bug caught by a spike in an error dashboard twelve minutes after a full deploy, with a rollback that takes another two minutes, is a production test I’m comfortable running weekly. The same bug shipped with no monitoring and no rollback path, discovered three days later from a customer complaint, is the actual thing “never test in production” should be warning you about, and it has nothing to do with which environment ran the code.