Slack’s engineering team discovered that flaky tests — tests that pass and fail on identical code — accounted for 56.76% of their CI failures. More than half of every build failure was a phantom, sending engineers chasing ghosts instead of shipping software. After investing in a dedicated remediation effort, Slack drove flakiness down to 3.85%, but the journey consumed months of engineering capacity that could have built features. The lesson was expensive and universal: the machinery built to catch bugs had itself become the bug.
Test automation has become one of the fastest-growing sources of technical debt in modern software development, yet it rarely appears on any team’s debt register. The suites that promised to reduce maintenance burden have themselves become the maintenance burden — demanding constant care, consuming compute resources, and delivering diminishing returns as they bloat beyond comprehension. The TestDino Flaky Test Benchmark Report 2026 found that the proportion of teams experiencing significant test flakiness grew from 10% in 2022 to 26% in 2025, a trajectory showing no signs of reversal.
The compounding cost
The financial damage is quantifiable. Developers spend up to 1.28% of their time repairing flaky tests, translating to approximately $2,250 per month per organisation in direct costs. For enterprise teams, more than 8% of total development time is consumed by fixing unreliable tests — time logged against feature tickets rather than quality budgets, making the true cost invisible to leadership. QA engineers spend an average of 20-30% of their work week simply triaging failures to distinguish between genuine regressions and flaky phantoms.
Microsoft’s research found that roughly 25% of test failures in large-scale CI systems are caused by flaky tests rather than actual code defects. Google’s testing infrastructure team has published extensively on the problem, documenting how flaky tests erode developer trust in the entire test suite — a second-order effect more damaging than the direct cost. When engineers learn to ignore red builds, real defects slip through undetected.
Test maintenance, including the fight against flakiness, now consumes roughly 40% of QA team time — none of it spent finding actual bugs. The World Quality Report 2025-26, published by Capgemini and Sogeti, found that 58% of quality engineering leaders described their automation ROI as “disappointing” or “unclear.”
The automation paradox
The irony runs deep. Automation was sold on the promise of reducing human toil, but poorly managed test suites create more of it. Engineering leaders at Google and Meta have moved away from exhaustive testing toward what the industry now calls “intent-based selection” and “autonomous healing” — strategies that acknowledge the old approach of running every test on every commit is economically unsustainable.
Spotify confronted this directly. The company audited its test suite and found that nearly a third of its automated tests were either redundant, permanently flaky, or testing code paths that no longer existed. The dead tests consumed thousands of compute hours weekly and triggered hundreds of false-positive alerts daily. Spotify deleted them — and saw zero increase in production incidents. The act of subtraction improved quality more than years of addition.
The pattern is consistent across the industry. A 2024 empirical study by researchers at Delft University of Technology, analysing over 18,000 test failures across open-source Java projects, found that flaky tests persisted in codebases for a median of 287 days before being fixed or removed — functioning as a slow tax on developer attention throughout their lifespan.
The tooling treadmill
The market’s response to automation debt has been to sell more automation. Playwright positioned itself as the answer to Selenium’s maintenance woes; Cypress had made similar promises earlier. Each tool genuinely reduced certain categories of flakiness, yet the State of JavaScript 2024 survey found that teams adopting Playwright still reported test maintenance as their primary pain point, ahead of initial setup, debugging, and CI integration. The fundamental problem is architectural, not tooling.
AI-assisted test generation — now offered by tools from Codium, Diffblue, and embedded in GitHub Copilot — threatens to accelerate the debt cycle rather than resolve it. These tools excel at producing tests quickly, but they encode current behaviour without distinguishing intended contracts from incidental implementation details. A team that uses AI to double its test count in six months will face quadrupled maintenance load within a year unless it simultaneously invests in test architecture, rigorous flakiness budgets, and the discipline to delete tests that no longer earn their keep.
The organisations that thrive will be those that treat their test suite not as an ever-growing archive but as a garden requiring constant, deliberate pruning. A lean suite that runs fast, fails meaningfully, and costs little to maintain will catch more real bugs than a sprawling one that drowns its signals in noise. The test suite is not a safety net — it is a tool, and like any tool, it works best when kept sharp.
References
- TestDino, Flaky Test Benchmark Report 2026: Rates, Root Causes, and Cost Implications, testdino.com
- StickyMinds, “The Hidden Costs of Flaky Tests: A Deep Dive into Test Reliability,” 2025, stickyminds.com
- Capgemini and Sogeti, World Quality Report 2025-26, capgemini.com
- Google Testing Blog, “Flaky Tests at Google and How We Mitigate Them,” testing.googleblog.com
- CloudQA, “Scaling Test Automation and Maintenance with AI-Driven Resilience,” 2025, cloudqa.io
Discussion