Coverage is the BMI of Software Quality: A Comfortable Lie Your Board Believes

Software engineers learned long ago that you cannot improve what you cannot measure. Boards took that lesson and ran with it, straight into the wrong metric. Code coverage (the percentage of source lines executed during a test run) now adorns dashboards from Silicon Valley to Singapore as the primary signal of software health. Mandate 80%, watch the number climb, congratulate the team, and file the slide. The ritual is reassuring, repeatable, and wrong. Code coverage is the BMI of software quality (a comparison Jared Toporek drew on the Stack Overflow blog, and it deserves to stick): a single number that feels scientific, correlates poorly with what actually matters, and is optimised above all else for its own improvement. Achieving a high score demands nothing so inconvenient as tests that catch bugs, merely tests that visit code. The board’s comfortable green dashboard may be the most expensive misunderstanding in modern engineering, and the industry has spent two decades building governance architecture on top of it.

A Correlation That Does Not Exist

The foundational claim behind every coverage mandate is that higher coverage means fewer bugs in production. Microsoft Research tested this assumption across 100 open-source Java projects and found that code coverage has an insignificant correlation with post-release defect counts at the project level and a file-level correlation statistically indistinguishable from zero in practical terms (about minus 0.02). The finding directly contradicts the premise on which engineering departments worldwide have built their quality regimes. What makes this finding particularly uncomfortable is the near-universal enforcement of the 80% threshold as though it carried scientific authority. According to the Stack Overflow engineering blog (December 2025), a search for scientifically tested evidence linking that threshold to software quality comes up empty; the figure appears to derive from a misapplication of the Pareto principle, dressed up in the language of rigour without any of its substance. The 80% mark is not a finding: it is a folk tale that has survived long enough to acquire the trappings of orthodoxy. Worse, the metric punishes good engineering. Toporek works a deliberately simple hypothetical: a 100-line file at exactly 80% coverage contains a ten-line block duplicated elsewhere. Consolidate the duplicate, as the DRY principle demands, and the file falls to 90 lines with 70 covered: 77.8%, below the gate. The numbers are illustrative, but the mechanism is not; he reports hitting it on a real bug fix, and the arithmetic holds for any refactoring that deletes covered lines faster than uncovered ones. Engineers who improve the codebase watch the score decline and must either reverse the improvement or pad the suite.

The Thirty-Four-Point Oracle Gap

If coverage tells you little, mutation testing tells you considerably more. The practice, introducing small, deliberate faults into source code and counting how many tests fail, exposes something coverage cannot: whether assertions actually validate behaviour, or merely observe that code ran without crashing. A practitioner study by J. Ghiringhelli, documented on Dev.to, found that a codebase reporting 93.1% line coverage yielded only a 58.62% mutation score. That 34-point gap represents the fraction of the codebase where a bug could be quietly introduced, all tests would continue to pass, and the CI pipeline would remain resolutely green. Three separate rounds of targeted assertion improvements were required to close it. The headline number had been accurate; it had simply been measuring the wrong thing: thoroughness of visitation, not rigour of judgement. Researchers at Carnegie Mellon and Northern Arizona University formalise this as the “oracle gap” (the difference between source-code coverage and mutation score), arguing it “surfaces important information about the extent and quality of a test effort beyond either adequacy metric alone,” particularly exposing files where weak assertions mask untested behaviour. A test that visits a function without asserting on its output is, in any meaningful sense, not a test. Coverage counts it; mutation testing does not. Google’s mutation-testing system, run by more than 24,000 developers across more than 1,000 projects in ten languages, shows the gap at industrial scale in the simplest way possible: mutants are generated only on lines that tests already cover, and some still survive. Every surviving mutant is a bug the green dashboard has already blessed.

The Perverse Machine

The problem is accelerating, and artificial intelligence is doing the accelerating. A study of 1.2 million 2025 GitHub commits (Hora and Robbes, MSR 2026) found that agents generate test commits with a 36% mock-commit ratio, compared with 26% for human developers. More revealing still is the narrowness: in 95% of repositories with agent mock activity, agents reached for the classic mock type, with spies (33%) and fakes (32%) far behind, whereas human-authored mock commits spread across mock (91%), fake (57%), and spy (51%). The effect within a given repository is statistically significant but small, which is exactly what you would expect this early. The study measures prevalence, not consequence; the inference is mine: tests built from a single isolation pattern are the cheapest kind to generate and the kind most likely to assert that code ran rather than that it worked. If that inference is right, coverage dashboards will inflate at machine speed. If agent-written suites turn out to kill mutants at human rates, this paragraph is wrong, and the data to check is already public.

The deeper problem is not the metric itself but the governance architecture built around it. When coverage becomes a deployment gate, a performance-review line item, or a board-level KPI, it stops being a signal and becomes a target. Goodhart’s Law, the observation by economist Charles Goodhart that any measure which becomes a target ceases to be a good measure, operates without mercy on software quality. Coverage was never a particularly good measure to begin with. Promoted to a target, it is actively harmful.

The honest counter-case deserves the floor. The same Google group found coverage genuinely useful as developer feedback (Code Coverage at Google, ESEC/FSE 2019), and its mutation service pointedly computes no mutation score at all: coverage decides which lines get mutated, and the authors call mutation adequacy neither practical nor desirable. Mutation score is itself an imperfect proxy with a contested link to real faults. The defensible claim is therefore narrower than “coverage lies”: coverage is a fine floor and a poor ceiling, and it is the gap between coverage and kill rate, not either number alone, that carries signal. What would prove this post wrong: codebases where high line coverage coincides with high mutation kill rates at scale, or evidence that closing the oracle gap fails to reduce escaped defects.

As the chart below illustrates, four numbers put the dynamic side by side: one team’s reported coverage, the folk-tale threshold, the same threshold after an honest refactoring, and what the tests actually caught.

The Monday version of this is checkable. Pick the most critical service in your fleet. Run a mutation testing tool (such as PIT for Java, Stryker for JavaScript/TypeScript/C#, or mutmut for Python) against its core domain logic. Measure two numbers: the reported line coverage versus the mutation kill rate. If the gap exceeds twenty percentage points, your test suite is asserting on presence, not behaviour, and your green build is an illusion you can measure before production surfaces it for you.

References

Microsoft Research. “Code Coverage and Post-Release Defects: A Large-Scale Study on Open Source Projects.” https://www.microsoft.com/en-us/research/publication/code-coverage-and-post-release-defects-a-large-scale-study-on-open-source-projects/
Ghiringhelli, J. “The AI Reported 93.1% Coverage. It Was 34% Off.” Dev.to, 2025. https://dev.to/jghiringhelli/the-ai-reported-931-coverage-it-was-34-290k
Jain, K., Kalburgi, G. T., Le Goues, C., and Groce, A. “Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts.” ASE, 2023. https://arxiv.org/abs/2309.02395
Petrović, G., Ivanković, M., Fraser, G., and Just, R. “Practical Mutation Testing at Scale: A View from Google.” IEEE Transactions on Software Engineering, 2021. https://dl.acm.org/doi/10.1109/TSE.2021.3107634
Toporek, J. “Making Your Code Base Better Will Make Your Code Coverage Worse.” Stack Overflow Engineering Blog, December 2025. https://stackoverflow.blog/2025/12/22/making-your-code-base-better-will-make-your-code-coverage-worse/
Hora, A. and Robbes, R. “Are Coding Agents Generating Over-Mocked Tests? An Empirical Study.” MSR, 2026 (analysing 2025 commits). https://arxiv.org/abs/2602.00409