The automation accountant: what test ROI looks like

Forrester’s 2025 Total Economic Impact analysis tracked 15 enterprise test automation programmes over three years and found that the median ROI was 143% — a strong return by any capital allocation standard. It also found that the median time to positive ROI was 23 months, that 4 of the 15 programmes never reached breakeven, and that the variance between the best and worst performers was a factor of twelve. Test automation ROI is real. It is also wildly unpredictable, poorly measured, and routinely misrepresented in the business cases used to justify it.

The measurement problem

Most organisations calculate test automation ROI using a formula inherited from vendor sales presentations: hours of manual testing replaced multiplied by the hourly cost of a tester. This arithmetic is seductive in its simplicity and misleading in its assumptions. It treats manual testing time as fully eliminable (it is not — exploratory testing, edge case investigation, and user acceptance testing remain irreducibly manual). It ignores the cost of building and maintaining the automation (which Capgemini’s 2025 report places at 40-60% of total automation effort). And it assumes that automated tests provide equivalent coverage to the manual tests they replace (which Gartner’s research disputes, finding that automated suites detect 30-40% fewer novel defects than skilled manual testers).

The organisations with accurate ROI measurements — the ones whose numbers hold up to CFO scrutiny — measure differently. They track total cost of quality: the sum of prevention costs (test development), appraisal costs (test execution), internal failure costs (bugs found before release), and external failure costs (production incidents). Automation shifts spending from appraisal to prevention. Whether that shift produces positive ROI depends on whether the prevented defects would have been more expensive to fix in production than the automation was to build.

Test automation ROI: the long road to payback — chart showing median ROI of 143% after 23 months, with wide variance across programmes Source: Forrester, Total Economic Impact of Enterprise Test Automation, 2025

The scale threshold

Automation ROI is not linear. Small-scale automation programmes — fewer than 500 automated tests — consistently produce negative or marginal returns, according to McKinsey’s 2025 analysis of 200 QA organisations. The fixed costs of tooling, infrastructure, and expertise required to maintain an automation programme create a breakeven threshold that small programmes cannot reach.

At Salesforce, the QA engineering team found that their automation programme became cost-positive only after reaching 2,000 automated tests — a milestone that took 14 months and required dedicated infrastructure that cost more than the testing tool licences. Below that threshold, the company would have been financially better served by hiring additional manual testers.

Above the threshold, returns compound. Netflix’s test engineering metrics show that each additional 1,000 automated tests reduces the marginal cost per test by 12%, as fixed infrastructure costs are amortised across a larger base. At scale, automation is unambiguously economical. The challenge is surviving the investment period before scale is reached.

The talent multiplier

The most underappreciated factor in automation ROI is the skill level of the engineers maintaining the suite. Deloitte’s 2025 Technology Workforce Report found that automation programmes staffed by engineers with more than five years of testing experience produced 3.2 times higher ROI than those staffed primarily by junior engineers or developers with testing as a secondary responsibility.

The explanation is not mysterious. Experienced test engineers write fewer but more targeted tests, design more maintainable frameworks, and diagnose failures faster. They understand which tests to automate and — equally important — which to leave manual. This judgment, which cannot be automated, is the single largest determinant of whether an automation programme delivers on its business case.

The honest forecast

Organisations initiating test automation programmes should budget for 18-24 months before breakeven, expect 15-25% efficiency improvement rather than the 50-70% that vendor presentations promise, and plan for automation maintenance to consume 40-60% of ongoing QA effort. These are realistic numbers that, if met, justify the investment. The programmes that fail are not the ones with modest returns. They are the ones that promised transformative returns and delivered modest ones — leaving stakeholders feeling deceived by an initiative that was, by any objective measure, a success.

Test automation efficiency: what gets promised vs delivered

Typical efficiency improvement · %

Vendor-claimed efficiency gain50–70%

Realistic efficiency improvement (Forrester median)15–25%

Sources: Forrester, Total Economic Impact of Enterprise Test Automation, 2025; Capgemini, World Quality Report, 2025

Understanding why automation costs escalate is as important as projecting returns. The hidden costs of test automation — infrastructure, maintenance overhead, and the opportunity cost of engineers maintaining brittle suites — routinely consume the efficiency gains that ROI models forecast. Capgemini’s 2025 report found that organisations underestimating these costs were three times more likely to report their automation programmes as unsuccessful, even when the programmes met their stated coverage targets. ROI is real. So is the risk of measuring it incorrectly.