AI code review tools: what they actually catch, and what they miss

Every pull request I open now gets a review from a tool before a human ever looks at it. That has been true for long enough that I stopped noticing it as new, which is exactly the point where it is worth actually checking whether it is earning that spot or just occupying it out of habit. So this is a plain accounting, a year of real pull requests, of what an AI review tool reliably catches and what it reliably does not.

What it actually catches, every time

The strongest category is anything checkable from the diff alone, with no knowledge of the rest of the system required. Unhandled error returns, a variable that shadows an outer one, a null check that got removed along with the line it was guarding, an obviously wrong comparison operator, a resource that gets opened but not closed on an early return path. These are the same bugs a careful linter would flag, except phrased in plain English with a suggested fix instead of a rule ID, which is a real usability improvement even when the underlying detection is not new.

It is also genuinely good at pattern-matching known security shapes: string concatenation going straight into a query, a secret that looks like it is being logged, user input reaching a shell call unescaped. These are exactly the kind of SQL injection and XSS shapes that have taught every scanner what to look for over the last two decades, and an AI reviewer is competitive with a dedicated static analysis tool here, sometimes better, because it can explain why the pattern is dangerous in the specific context of that line instead of just naming the rule.

What it misses, consistently

Everything that requires understanding the system beyond the diff is where it falls apart. A loop that issues one query per row looks completely unremarkable in isolation, since nothing about that single file says how many rows the caller actually passes in production. I wrote about exactly this shape of bug after it took down a connection pool at four thousand rows, and no AI review flagged it beforehand, because flagging it requires knowing the call site’s real data volume, not just reading the loop.

Business logic correctness is the other consistent miss. A function that computes a discount, applies a tax rule, or checks a permission can be syntactically flawless and still be wrong in a way only someone who knows the actual business rule would catch. The tool has no way to know what the code was supposed to do, only whether what it does looks internally consistent, and those are not the same question. Concurrency bugs land in the same place, a race condition or a missing lock is often invisible from a static diff and only shows up under real, timed, concurrent load, which a text-based review has no way to simulate.

The false positive tax is real

The part that gets talked about less is the cost of the comments that are technically correct but not actually worth fixing, style suggestions restated as if they were bugs, or a “consider adding error handling here” on a code path that cannot actually fail given the caller. Enough of those in a row and reviewers start skimming past AI comments as a category, which is the worst outcome, because the next comment might be the one that actually matters. Keeping the tool useful has meant actively tuning what it is allowed to comment on, not just accepting its default settings.

Where it actually belongs

The honest framing, after a year of this, is that it is a fast first pass that clears the mechanical layer off a human reviewer’s plate, not a substitute for someone who understands the system reviewing the change. I still read every diff myself, but I am reading it for the things the tool structurally cannot see, whether this is the right change to make, not just whether the fifteen lines in front of me are internally correct. That split has held up. The moment I have caught myself skipping a real review because the tool already said the PR looked fine is the moment I have gotten burned, always by something that was never in scope for it to catch in the first place.