Blog
Flaky Tests: Causes, Detection, and How to Fix Them

Abhilash
Published on

Table of contents
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
Lorem ipsum
A flaky test is an automated test that passes sometimes and fails other times, even though nothing has changed. Same code, same test, same environment, same inputs, and you still get a different result.
That definition is narrow on purpose. If a test fails because the app is broken, it isn’t flaky. It’s doing its job. If it fails because someone changed the test and got it wrong, that’s a bug in the test. A test is only flaky when neither the app nor the test has changed and the result still flips.
Flaky tests are a problem everywhere, even at companies with huge engineering teams. In 2016, Google shared that about 1.5% of its test runs were flaky, that almost 16% of its tests showed some flakiness, and that around 84% of tests that went from passing to failing were down to flakiness, not real bugs.
The real damage isn’t one test lying now and then. It’s that people stop believing the test results at all. When results keep flipping, every red build has to be checked twice: once to work out if it’s real, and again to fix it if it is. That double effort is what turns flaky tests from an annoyance into a serious problem.
Why Even a Few Flaky Tests Break a Big Suite
It’s tempting to think a 1% or 2% flaky rate is fine. But small rates add up fast once you have lots of tests.
Tests in the Suite | Chance Each Test Passes | Chance the Whole Suite Goes Green |
|---|---|---|
100 | 99.95% | About 95% |
500 | 99.95% | About 78% |
1,000 | 99.95% | About 61% |
Even if each test only flakes once in 2,000 runs, a 1,000-test suite will fail for no real reason around four times out of ten. That’s why you can’t handle flaky tests one at a time in a big suite.
You need to track flakiness across the whole suite as a proper metric.
What Flaky Tests Really Cost You
The obvious cost is time spent rerunning and rechecking tests. The bigger costs are harder to see.
Cost | What Happens |
|---|---|
People stop trusting tests | Engineers get into the habit of hitting “retry” first and only looking into a failure if it happens again. Real bugs slip through that way |
Slower pipelines | Retries add time to every run, and across a large team committing several times a day, that can add up to many hours a month |
Coverage looks better than it is | Flaky tests still count towards coverage, even though nobody trusts what they say |
More bugs reach production | A real failure gets written off as “just flaky” and makes it into a release. One incident like that can cost more than a year of retries |
Frustrated engineers | Automation engineers end up chasing random failures instead of building anything, which wears people down |
The 7 Main Causes of Flaky Tests
Most flaky tests come down to one of seven causes, and each one needs a different fix. Timing comes up the most.
A 2014 study of flaky test fixes in open source projects found about 45% were caused by tests not waiting properly for async work to finish, so it’s the first thing to check.
Cause | What Goes Wrong | How to Fix It |
|---|---|---|
Timing and async behaviour | The test assumes a page has loaded, an animation has finished or an API has replied, and sometimes it hasn’t. Fixed sleeps hide the problem for a while | Wait for something specific, like an element becoming clickable or a request finishing, instead of a set time |
Fragile element locators | Tests find elements with deep CSS selectors or XPath, which break whenever the page structure changes, even if nothing a user would notice has changed | Find elements by their label, role or position on the page, and use tools that suggest fixes when the page changes |
Test data problems | Tests rely on data from other tests or leftovers from earlier runs, so they fail when the data isn’t what they expect | Have each test create its own data, run against a known state and clean up after itself |
Unstable environments | A slow database, a patchy network, a third-party timeout or an environment redeployed mid-run | Keep test environments close to production, separate concurrent runs, and handle outside failures cleanly |
Tests running in a different order | Tests that share state clash when they run in parallel or in a different order | Make every test independent, so it can run in any order on any machine |
Third-party services | An outside service has a bad day, and your test fails even though your app is fine | Mock outside services in unit and integration tests, and test against the real ones on a set schedule |
Results that change by design | Random IDs, timestamps, unsorted lists or AI-generated text mean exact checks fail no matter how well the test is written | Check properties that stay true, like a valid ID format or a time within a range, instead of exact values |
Here’s what the timing fix looks like in practice. This version waits a fixed time and hopes for the best:
This version waits for the thing it actually needs:
And here’s the fix for results that change by design. Instead of checking an exact order ID, check that it looks like a valid one:
Flaky Tests vs Tests That Always Fail
Both show up as a red build, but they’re very different problems.
Test That Always Fails | Flaky Test | |
|---|---|---|
Can you reproduce it? | Yes, every time | Not reliably |
Is the result honest? | Yes, something really is wrong | No, it fails without a real reason |
How hard is it to fix? | Usually straightforward | Hard, because you can’t make it fail on demand |
What it does to the team | Points at a real problem | Teaches people to ignore red builds, including real ones |
What to do | Fix it straight away | Track it, move it aside and investigate it properly |
How to Find Flaky Tests
Flaky tests don’t label themselves, so you have to look at results over many runs to spot them.
Look at Pass and Fail History
A test that usually passes but occasionally fails, with no related code change, is a strong candidate. Agree on a threshold that makes sense for your team, rather than relying on a fixed industry number.
Track How Often Retries Pass
If your CI retries failed tests, a test that keeps passing on the second try is very likely flaky. Retries hide the problem, so log them and keep an eye on the numbers.
Check Where and When it Fails
A test that fails on one CI machine but not others, or only when the system is busy, is reacting to the environment, not the code.
Run it Lots of Times
Run a suspect test 50 or 100 times on the same build. If the results vary, it’s flaky.
Watch the Trend
One odd failure might be a one-off. The same pattern over weeks is flakiness. A chart of your flaky rate over time catches what single reports miss.
How to Manage Flaky Tests: A Six-Step Process
Finding flaky tests isn’t enough on its own. You need a clear process for dealing with each one.
Step | What Happens |
|---|---|
Detect | Flaky tests get flagged automatically based on failure rate, retry results and where they fail, instead of engineers noticing the same retries over and over |
Quarantine | The test moves out of the suite that blocks merges. It still runs and reports, but it can’t hold up a release. Quarantine is a waiting room, not a permanent home |
Diagnose | Someone is given time to work out the cause. This is the most expensive step and the one teams skip most, which is how quarantine fills up |
Fix or delete | Fix the test based on its cause, or delete it if it’s no longer worth keeping. A smaller suite you trust is better than a bigger one you don’t |
Check the fix | Run the fixed test many times on the same build. Passing once doesn’t prove anything |
Bring back or retire | Tests that stay stable go back into the main suite. Tests that don’t go back to diagnosis or get retired |
Fewer flaky tests also makes it easier to decide which tests to run on each change. Our guide to risk-based regression testing covers how to choose which tests to run based on what changed.
Tools for Spotting Flaky Tests
Option | How It Helps |
|---|---|
CI platform features and plugins | Several CI tools can flag flaky tests, either built in or through plugins. Azure DevOps and CircleCI have built-in flaky test reporting, and Jenkins has plugins for it. Check what your own CI tool offers |
Dedicated flaky test tools | Track results across runs and spot patterns that simple retries miss. Many read JUnit, pytest and Jest results directly |
Your own dashboards | Bigger teams often build their own, so they can link flakiness to deployments, environment changes and code owners |
AI failure analysis | AI can look at failure patterns, timing and retry results to tell flaky failures apart from real but rare bugs |
How to Stop Flaky Tests Before They Start
Detection deals with flaky tests after they show up. Prevention stops them showing up in the first place.
Wait for Things, Not for Time
Every wait should be for something specific in the app, not a fixed number of seconds. A Thread.sleep is a warning sign.
Describe What the User Does, Not How the Page is Built
Tests written around user actions survive redesigns much better than tests tied to today’s HTML.
Use Tools That Suggest Fixes When the Page Changes
When an element moves or gets renamed, a good tool finds the new one and proposes an update for someone to check, instead of failing or silently changing the test.
Look After Your Test Data
Fresh, versioned or synthetic data means tests always start from a known state.
Keep Tests Independent
Tests that can run in any order on any machine are safe to run in parallel.
Mock Unreliable Outside Services
Test your app separately from the services it depends on, and test the real services on a schedule.
Check Properties, Not Exact Values
Where results change by design, check the format, range or allowed values.
Independent, stable tests are also what make parallel runs safe. Our guide to parallel test execution explains how to run tests side by side without them tripping over each other.
Flaky Tests and AI-Written Code
AI now writes a lot of the code in many teams, and that makes flaky tests a bigger deal in three ways:
More code means more test runs: More pull requests and deployments mean tests run more often, so flaky failures happen more often too.
More refactoring means more broken locators: AI tools rewrite code regularly, changing page structures and IDs faster than many test suites can keep up with.
Less patience for noise: If you ship every day, you can’t afford a test suite your team has stopped believing.
None of this makes flaky tests acceptable. It means the habits that prevent them, like waiting for real events, describing user actions, looking after data and keeping tests independent, are now essential rather than nice to have.
How Virtuoso QA Cuts Down Locator-Driven Flaky Tests
Most causes of flaky tests need careful, case-by-case work. Broken locators are one of the biggest causes in UI test suites, and they’re also the one AI-written code makes worse, because every refactor shuffles the page structure.
That’s the cause Virtuoso QA is built to tackle.
Tests that point at what users see: A Virtuoso QA step like “click Submit order” finds the button by its label and surroundings, not a long CSS path. If a developer renames a class or wraps the button in a new div, the test doesn’t care.
A heads-up instead of a red build: If a page change does trip a step up, Virtuoso QA works out which element the step was after and suggests the update. Nothing changes in the test until someone on your team approves it, and the decision is logged, so a real bug can’t be quietly patched over as flakiness.
Built-in waiting: Each step waits for the page to be ready before acting, so you don’t end up with
Thread.sleepcalls scattered through your tests.Same result every run: Tests run on a deterministic engine, so if the app and the test haven’t changed, the outcome shouldn’t either. When results do differ, that’s a strong sign something in the environment or data is off.
Clues when something fails: The failed step shows up next to the last passing run of the same step, with screenshots and logs, so it’s quicker to tell a flaky failure from a real one.
Stable tests are what make regression worth running on every change. To see how Virtuoso QA keeps regression suites dependable through constant releases, take a look at regression testing.
Related Reads
Flaky Tests FAQs
What is a flaky test?
What causes flaky tests?
How do you find flaky tests?
Should you just retry flaky tests?
What does quarantining a flaky test mean?
How do you stop flaky tests from happening?
Written by

Abhilash
Industry Analyst, Test Automation
Abhilash tracks how AI is reshaping software testing, from pairwise coverage to regulatory accountability, and writes research-led pieces that cut through vendor noise with numbers.












