Google’s testing infrastructure processes over four billion test executions per day across a codebase exceeding two billion lines. The company enforces no universal coverage target. Not 80%, not 90%, and certainly not 100%. Instead, Google’s engineering productivity team recommends that teams pursue “useful coverage” — tests that have historically detected real defects — and explicitly warns against treating coverage percentages as quality proxies. When the company with the most sophisticated testing infrastructure on Earth declines to mandate a coverage floor, it is worth asking what everyone else thinks they know that Google does not.
The thesis is blunt: test coverage, measured as a percentage of lines or branches exercised, has become the most widely gamed metric in software engineering. Teams chase coverage numbers not because high coverage reliably prevents defects but because the number is easy to measure, easy to mandate, and easy to present to executives who want a simple quality score. The result is a perverse incentive structure that rewards writing trivial tests and penalises thoughtful engineering judgement.
The correlation that isn’t
The empirical case against coverage targets is damning. Laura Inozemtseva and Reid Holmes at the University of Waterloo examined the relationship between coverage and test-suite effectiveness across five large Java programmes. After controlling for test-suite size, they found that coverage had low to moderate correlation with fault detection. Statement coverage, branch coverage, and even modified condition/decision coverage explained far less variance in bug-finding ability than the sheer number and diversity of test cases. Stronger forms of coverage did not provide greater insight into suite effectiveness.
The finding has been replicated and extended. A 2024 ACM study confirmed that mutation coverage — a supposedly more rigorous metric — is itself not strongly correlated with mutation coverage across different granularities, undermining the assumption that any single metric captures test quality. The research converges on an uncomfortable conclusion: coverage measures whether code was executed during testing, not whether behaviour was verified.
Martin Fowler, ThoughtWorks chief scientist, framed it plainly: “I would say you are doing enough testing if the following is true — you rarely get bugs that escape into production, and you are rarely hesitant to change some code for fear it will cause production bugs.” Coverage appears nowhere in that definition.
The gaming epidemic
Coverage mandates create predictable pathologies. When teams face an 85% or 90% threshold, they meet it by the fastest available means. Codecov’s data confirms the pattern: the simplest code — getters, setters, configuration boilerplate — typically achieves the highest coverage, while complex business logic where bugs actually cluster sits well below the codebase mean. Teams inflate the aggregate by testing trivia while leaving risk untested.
The problem extends beyond lazy testing. AI-assisted test generation tools — GitHub Copilot, Diffblue Cover, CodiumAI — now make it trivially easy to generate tests that inflate coverage numbers. An engineer can achieve 95% coverage in minutes with auto-generated assertions that verify return types and null checks without testing meaningful behaviour. The metric improves; the quality does not. As a Trail of Bits engineering blog post in September 2025 argued, these tools accelerate the production of “assertion-free tests” — tests that touch code, increment the counter, but verify nothing about its behaviour.
Mutation testing’s quiet revolution
The alternative is already in production at companies willing to measure quality rather than activity. Mutation testing — which deliberately introduces small faults into code and checks whether the test suite detects them — provides a direct measure of test effectiveness. Goran Petrovic and Marko Ivankovic documented Google’s mutation testing infrastructure in an IEEE paper, demonstrating that high-coverage code routinely contained “surviving mutants” — seeded bugs that no test caught. A codebase at 95% line coverage might detect only 60% of seeded mutations, exposing a 35-point gap between the coverage metric and actual fault-detection capability.
IEEE research has shown that mutation testing catches 10-15% more defects than traditional coverage-based approaches. Pitest, the open-source Java mutation testing framework, has seen its adoption triple in recent years. StrykerJS, its JavaScript counterpart, reports similar growth. The shift is slow but directional: the industry is gradually discovering that the question worth asking is not “did my tests run this code?” but “would my tests catch a bug here?”
The executive dashboard problem
Coverage persists as a target because it fills a communication vacuum. Executives need a number to assess quality, and coverage is the number testing teams have always given them. Replacing it requires offering something better — and mutation scores, defect escape rates, and mean-time-to-detection are harder to explain in a quarterly review.
The engineering leaders who successfully retire coverage mandates reframe quality in business terms: defects reaching customers, revenue lost to incidents, engineering hours burned on rework. These metrics are harder to game precisely because they measure outcomes rather than activity. The coverage number will continue to appear on dashboards for years. But its authority is eroding, test by meaningless test, as the companies that take quality seriously learn to measure what matters instead of what is easy.
References
- Inozemtseva, L. and Holmes, R., “Coverage Is Not Strongly Correlated with Test Suite Effectiveness,” Proceedings of ICSE 2014, researchgate.net
- Petrovic, G. and Ivankovic, M., “State of Mutation Testing at Google,” Proceedings of ICSE-SEIP 2018, IEEE
- Codecov, “Mutation Testing: How to Ensure Code Coverage Isn’t a Vanity Metric,” 2025, codecov.io
- Trail of Bits, “Use Mutation Testing to Find the Bugs Tests Miss,” September 2025, blog.trailofbits.com
- ACM, “Mutation Coverage is not Strongly Correlated with Mutation Coverage,” Proceedings of AST 2024, dl.acm.org
Discussion