Blog
Flaky Tests: Causes, Detection, and How to Fix Them
Published on
August 19, 2026
Pricing
A flaky test is an automated test that produces inconsistent results across executions without any change to the code under test or the test itself.
A flaky test is an automated test that produces inconsistent results across executions without any change to the code under test or the test itself. The same test, run on the same build, against the same environment, with the same inputs, sometimes passes and sometimes fails.
The definition is precise on purpose. A test that fails because the application is broken is not flaky, it is correctly reporting a defect. A test that fails because it was updated incorrectly is not flaky, it has a bug. Flakiness is reserved for the case where neither the test nor the application has changed and yet the outcome varies.
The scale of the problem is well documented. Google's published research found that around 1.5% of test runs at Google exhibit flaky behaviour, affecting nearly 16% of their tests, and strikingly that when a test transitions from passing to failing in their post-submit system, around 84% of the time it is a flaky result rather than a real regression.
The flaky rate is not zero in any organisation operating at meaningful scale, and the danger is not that a single test occasionally lies. The danger is that the whole suite becomes hard to read, because when pass rates oscillate, every failure has to be triaged twice, once to decide whether it is real and again to fix the actual defect if it is. That double cost is what makes flakiness a strategic problem rather than a tactical annoyance.
The instinct is to dismiss a 1% or 2% per-test flake rate as tolerable. The arithmetic says otherwise, because flakiness compounds across a suite.
Suppose each test independently passes 99.95% of the time, a flake rate of just 0.05%. A suite of 100 such tests runs completely green only about 95% of the time, since 0.9995 to the hundredth power is roughly 0.95. Scale that same tiny per-test rate to 1,000 tests and the suite passes cleanly only about 61% of the time, since 0.9995 to the thousandth power is roughly 0.61.
So a per-test flake rate low enough to look harmless means a large suite fails a green run around four times in ten, for no real reason at all.
The lesson is that flakiness cannot be managed at the level of the individual test in a large estate. A rate that is invisible per test is catastrophic per suite, which is why serious programmes treat the suite-level flaky rate as a first-class operational metric rather than shrugging off the odd intermittent failure.
The visible cost of flaky tests is the time spent re-running and re-triaging them. The hidden costs run deeper and do most of the damage.
When tests fail intermittently, engineers stop treating failures as signals and develop a reflex to retry first and investigate only if the retry also fails. That reflex catches real regressions less often than the team likes to believe, and with 84% of pass-to-fail transitions being flaky at Google's scale, the reflex is understandable and dangerous in equal measure.
Retries lengthen execution, so a 30-minute regression run that retries 5% of failures grows towards 45 minutes, and across a hundred developers committing several times a day the cumulative loss runs into hundreds of engineering hours a month.
Flaky tests count toward coverage even when nobody trusts the result, so a team with 90% coverage and a 7% flaky rate effectively has 83% of its coverage producing reliable signal, and the headline number flatters the real state of the suite.
The most expensive cost is the regression that slips through because the failure was assumed to be flakiness. A single production incident traced back to a real failure that was retried away can outweigh a year of retries combined.
SDETs who joined to build automation end up debugging intermittent behaviour, work that is reactive, repetitive, and rarely rewarded, and sustained flakiness corrodes morale in exactly the engineers who would otherwise be improving the platform.
Most flaky tests trace back to one of seven causes, and naming the cause is the first step to fixing it, because the remediation is different for each. Timing dominates the list. Academic analysis of flaky-test fixes found that roughly 45% of flakiness comes from async-wait issues alone, which is why the first cause below is the one to check first.
The most common cause by a wide margin. Tests that assume a page has finished loading, an animation has completed, or an API has responded fail intermittently when the timing varies, and hard-coded sleeps mask the problem temporarily without solving it.
The fix is to replace fixed waits with conditional waits that key off explicit application state, namely an element becoming clickable, a request completing, or a value appearing in the DOM.
Tests that target elements by deep CSS selectors or XPath break when the DOM changes, and the change need not be functional, since a refactor that renames an internal class is enough to invalidate the locator.
The fix is to identify elements by intent, namely the visible label, the role, or the position relative to other elements, with self-healing further reducing this category by repairing locators automatically.
Tests that depend on data created by other tests, left over from previous runs, or reset unpredictably fail when the data state is not what they expect.
The fix is to treat test data as a managed resource, so each test sets up the data it needs, runs against a known state, and cleans up after itself.
Tests run against shared environments inherit every issue of those environments, namely a slow database, a flaky network, a third-party API that times out, or an environment redeployed mid-run.
The fix is environment hygiene, namely parity with production, isolation between concurrent runs, and clean failure modes when external systems misbehave.
Tests that share state and run in parallel fail when they execute in a different order than during authoring, producing race conditions that are hard to diagnose.
The fix is strict test independence, so tests run in any order, on any node, without side effects on each other.
Calls to third-party services introduce flakiness whenever the third party has a bad day, and the test fails not because the application is broken but because the dependency is.
The fix is contract testing with mocked dependencies at the unit and integration layer, plus periodic full-stack runs against real dependencies on a controlled cadence.
Some behaviour is intentionally non-deterministic, namely random IDs, timestamps, unsorted list ordering, or model outputs, and tests that assert on the exact output fail regardless of how well they are written.
The fix is to assert on properties that hold across runs, namely that the ID is a valid UUID or the timestamp is within a window, rather than on specific values.
It is worth drawing a distinction that changes how a failure should be handled, namely between a flaky test and one that fails consistently. They feel similar in the moment, a red build either way, but they are opposite problems.
A consistently failing test is, counterintuitively, the easier case. It fails the same way every time, so it can be reproduced, diagnosed, and fixed once, and its signal is honest, namely that something is truly wrong with the test or the application.
A flaky test is harder precisely because it does not fail reliably, so it cannot be reproduced on demand, it resists diagnosis, and worst of all it teaches the team to distrust every red build including the honest ones.
The practical implication is that a consistently failing test should be fixed on sight, while a flaky test needs a lifecycle, namely detection, quarantine, and deliberate diagnosis, because there is nothing to fix on sight when the failure will not reliably reappear.
A flaky test does not announce itself, so identifying flakiness needs instrumentation, namely data on test outcomes across many runs analysed for the patterns that distinguish intermittent behaviour from real failure.
The simplest indicator is a test that passes most of the time and fails occasionally with no correlated code change, and a failure rate below 2% on stable code is the conventional cut-off for classifying a test as flaky.
Where CI retries failed tests, the retry success rate is the cleanest single indicator, since a test that passes on retry far more often than expected is producing flaky behaviour. Retries hide the symptom, so logging them and surfacing the rate exposes flakiness that would otherwise be invisible.
A test that fails on one CI node but passes on others, or fails under peak load but passes when quiet, is sensitive to environment rather than code, and environmental correlation is one of the most predictive signals of flakiness.
A suspected test can be confirmed by running it many times on the same build in the same environment, and if it produces inconsistent results across a hundred runs it is flaky regardless of what any single execution showed.
A single intermittent failure may be a one-off, while a pattern across weeks is flakiness, so a dashboard showing flaky ratio over time catches what point-in-time reports miss.
Detection without management leaves the team in the same position with more data. Effective management runs a clear lifecycle on every flaky test, with each stage producing a definite outcome.
Detection logic runs continuously, classifying tests by failure rate, retry pattern, and environmental sensitivity, so flaky tests are flagged automatically rather than found by engineers stumbling on the same retries repeatedly.
A flagged test moves out of the blocking regression suite into a quarantine set that still executes and reports but does not block merges or releases, which removes the noise from the main signal while preserving visibility. Quarantine is a working state, not a destination, and a test that lives there indefinitely is dead weight.
The team allocates dedicated time to investigate quarantined tests, classify them by root cause, and decide on remediation. Diagnosis is the most expensive stage and the most often skipped, and programmes that quarantine without diagnosing accumulate technical debt in slow motion.
Each test is either fixed with the remediation specific to its root cause or deleted when it no longer earns its place, and the asymmetry matters, since a smaller reliable suite is worth more than a larger unreliable one.
A remediated test is re-executed many times against the same build to confirm the flakiness is actually gone, because a test that passes once after a fix is not yet ready to rejoin the trusted suite. Stability under repeated execution is the bar.
Tests that pass verification rejoin the active suite, and tests that fail it go back to diagnosis or get retired, which closes the loop.
Tooling for flaky test detection has matured in the last five years, and the choice usually comes down to integration with the existing pipeline and the size of the estate.
Detection and management deal with flakiness after it appears. Prevention deals with the conditions that produce it, and programmes that invest in architectural prevention reduce the rate at source rather than chasing it test by test.
Timing is the single biggest source of flakiness, so every wait should key off an explicit application state rather than an arbitrary duration, and fixed sleeps are a smell.
Tests that describe what the user does are more durable than tests that describe how the DOM is shaped today, and intent-based authoring is the structural reason some estates survive UI refactors and others do not.
Self-healing recognises that an element has changed, identifies the new one, and updates the test without human intervention, which removes locator-driven flakiness, consistently the largest single category in enterprise suites.
Synthetic, refreshable, versioned data removes the variability that produces data-driven flakiness, so tests run against known states and cleanup is deterministic.
Tests that run in any order on any node without side effects make parallel execution safe, and concurrency flakiness disappears at the architectural level.
Mocking third parties at the unit and integration layer separates the test of the application from the test of the dependency, with real dependencies exercised on a controlled cadence rather than every commit.
Where output is non-deterministic, assertions should check the property, namely valid format, within range, or in the expected set, which removes a whole class of false positives.
AI assistants now write a substantial portion of new code in enterprise environments, and the shift has three consequences for flakiness.
The implication is not that flakiness becomes acceptable. It is that the architectural choices that absorb flakiness, namely intent-based authoring, self-healing, conditional waits, test data hygiene, and independence, become non-optional rather than nice-to-have.
Most of the seven root causes need disciplined, case-by-case management. One of them does not have to. Locator-driven flakiness is consistently the largest single category in enterprise estates, and it appears wherever tests target elements by implementation detail and the implementation changes underneath them. It is also the category AI-generated code inflames most, since frequent refactors churn the DOM.
The architectural fix is intent-based identification combined with self-healing, and Virtuoso QA is built around both. Three mechanisms work together on the locator category specifically.
Try Virtuoso QA in Action
See how Virtuoso QA transforms plain English into fully executable tests within seconds.
