A software engineer who dismisses a failing CI pipeline as “probably flaky” is not cutting corners—she is making a statistically well-informed decision. That quiet act of professional resignation, multiplied across thousands of engineers and hundreds of daily builds, is how flaky tests drain engineering budgets: not with a sudden catastrophe but with the gentle persistence of a dripping tap that nobody has been asked to fix. The argument here is blunter than most engineering post-mortems allow: flaky tests are not a nuisance to be managed with auto-retry scripts but a quantifiable payroll cost, a signal-to-noise crisis, and a structural defect in how software quality is measured. The numbers, once examined, make this case almost without assistance.
The Payroll Cost of Phantom Failures
At Atlassian, flaky tests waste an estimated 150,000 developer-hours per year in the Jira backend repository alone, according to Atlassian’s engineering blog. In the Jira Frontend repository, 21% of all master build failures are caused not by defective code but by tests that disagree with themselves from one run to the next—meaning one in five emergency Slack notifications is, strictly speaking, a lie. No other category of engineering waste operates so openly and at such scale with so little executive scrutiny. A peer-reviewed industrial case study presented at ICST 2024, tracking 30 engineers over five years, put the broader cost in terms that any CFO can read without a tutorial in continuous integration. Flaky tests consumed at least 2.5% of productive developer time: 1.1% investigating false failures, 1.3% repairing them, and the remainder maintaining detection tooling. Across the study period, that amounted to roughly 6,600 developer-hours—equivalent to 3.75 full developer-years evaporated not on shipping features or closing security vulnerabilities, but on chasing noise.
A Signal-to-Noise Catastrophe
Google’s engineers discovered that 16% of all tests in the company’s codebase carry some degree of flakiness, a figure remarkable enough on its own. More corrosive is what Google found when examining build transitions: 84% of test-suite transitions from passing to failing are caused by flaky tests rather than genuine code defects, according to Google’s Testing Blog. As the chart below illustrates, when a build turns red, the overwhelmingly probable explanation is not that a developer introduced a regression—it is that a test decided to behave differently on a Tuesday. Engineers, being empiricists, eventually update their priors accordingly and stop trusting the signal. This erosion of trust is the compounding cost that balance sheets do not capture. An alert system that cries wolf 84% of the time is not an alert system; it is ambient noise. Teams that have learnt to distrust their test suites begin shipping with lower confidence in quality gates, schedule additional manual review sessions, and—most damagingly—delay deployments until someone with institutional knowledge can intuit whether a failure “looks real.” The hidden overhead has been quietly offloaded from the pipeline to the engineer’s nervous system.
The Arithmetic of Investigation
The ICST 2024 case study also produced one of the starkest pieces of unit economics in recent software engineering research: manual investigation of a flaky pipeline failure costs $5.67 in developer time, versus $0.02 for an automatic rerun—a 280-fold gap. This arithmetic explains why silent auto-retry has become the de facto industry response. It is cheaper, in the narrow sense, to rerun a test three times and declare it stable than to pay a senior engineer to diagnose why it failed. The trouble is that auto-retry is an anaesthetic, not a cure; it masks the systemic rot while the cost accumulates elsewhere. Microsoft researchers found that flaky test failures could be reproduced in only 25–43% of cases even after 500 individual test runs under controlled conditions, per analysis reported by CloudBees. When the failure cannot be reliably reproduced, root-cause analysis becomes not merely expensive but epistemically impossible—the engineer is hunting a ghost. A large fraction of flaky tests are therefore never fixed; they are quarantined, muted, or deleted, taking whatever signal they once carried with them into the void.
Asynchronous Roots, Concentrated Fixes
The news is not uniformly grim. A 2025 empirical study of 22 Java projects, published on arXiv, found that 75% of flaky tests cluster into co-failing groups rather than failing in isolation, which means failures are correlated and not random. More usefully, nearly half of all flakiness—45%—traces to a single root cause: asynchronous wait errors. Tests that race against timers, threads, and I/O operations, and lose, account for the lion’s share of the entire problem. This concentration is good news for engineering teams willing to invest in a targeted fix rather than a general retry policy; a single focused sprint on async test hygiene could halve the noise that is currently drowning the signal. Engineering teams that treat flakiness as background radiation—something to be tolerated rather than measured, bounded, and eliminated—are paying a recurring tax on every deployment they ship. The most interesting question is not whether flaky tests are expensive; the arithmetic above settles that. It is why so many engineering leaders have tacitly agreed to keep paying. Auto-retry is a deal struck with short-term incentives: it makes the dashboard green, the standup short, and the quarterly metrics tidy. What it does not do is stop the clock on those 6,600 developer-hours quietly ticking away in the background. The dripping tap has a way of flooding the basement—and by then, the plumber charges considerably more than the pipe was worth.
References
- Parry, J. et al. “Cost of Flaky Tests in CI: An Industrial Case Study.” ICST 2024 Industry Track, April 2024. https://conf.researchr.org/details/icst-2024/icst-2024-industry/1/Cost-of-Flaky-Tests-in-CI-An-Industrial-Case-Study
- Atlassian Engineering. “Taming Test Flakiness: How We Built a Scalable Tool to Detect and Manage Flaky Tests.” Atlassian Engineering Blog. https://www.atlassian.com/blog/atlassian-engineering/taming-test-flakiness-how-we-built-a-scalable-tool-to-detect-and-manage-flaky-tests
- Micco, J. “Flaky Tests at Google and How We Mitigate Them.” Google Testing Blog, May 2016. https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
- CloudBees Engineering. “The Flaky Test Confession: Ignoring Test Failures.” CloudBees Blog. https://www.cloudbees.com/blog/the-flaky-test-confession-ignoring-test-failures
- arXiv (2025). “Empirical Study of Flaky Test Co-Failure Groups in 22 Java Projects.” https://arxiv.org/html/2504.16777v1
Discussion