Platform

Solutions

Resources

Platform

Solutions

Resources

Platform

Solutions

Resources

Blog

Flaky Tests: Causes, Detection, and How to Fix Them

Abhilash

Published on

Flaky Tests
eBook CTA Background
RECOMMENDED RESOURCES

The Ultimate Enterprise Guide To Agentic Testing

AI agents can write, fix and triage your tests at scale. See how to use them in QA while keeping people in control of everything that ships.

Get the eBook →

Enterprise guide to agentic testing
eBook CTA Background
RECOMMENDED RESOURCES

The Ultimate Enterprise Guide To Agentic Testing

AI agents can write, fix and triage your tests at scale. See how to use them in QA while keeping people in control of everything that ships.

Get the eBook →

Enterprise guide to agentic testing
eBook CTA Background
RECOMMENDED RESOURCES
Enterprise guide to agentic testing

The Ultimate Enterprise Guide To Agentic Testing

AI agents can write, fix and triage your tests at scale. See how to use them in QA while keeping people in control of everything that ships.

Get the eBook →

eBook CTA Background
RECOMMENDED RESOURCES

The Ultimate Enterprise Guide To Agentic Testing

AI agents can write, fix and triage your tests at scale. See how to use them in QA while keeping people in control of everything that ships.

Get the eBook →

Enterprise guide to agentic testing

Table of contents

Lorem ipsum

Lorem ipsum

Lorem ipsum

Lorem ipsum

Lorem ipsum

A flaky test is an automated test that passes sometimes and fails other times, even though nothing has changed. Same code, same test, same environment, same inputs, and you still get a different result.

That definition is narrow on purpose. If a test fails because the app is broken, it isn’t flaky. It’s doing its job. If it fails because someone changed the test and got it wrong, that’s a bug in the test. A test is only flaky when neither the app nor the test has changed and the result still flips.

Flaky tests are a problem everywhere, even at companies with huge engineering teams. In 2016, Google shared that about 1.5% of its test runs were flaky, that almost 16% of its tests showed some flakiness, and that around 84% of tests that went from passing to failing were down to flakiness, not real bugs.

The real damage isn’t one test lying now and then. It’s that people stop believing the test results at all. When results keep flipping, every red build has to be checked twice: once to work out if it’s real, and again to fix it if it is. That double effort is what turns flaky tests from an annoyance into a serious problem.



Why Even a Few Flaky Tests Break a Big Suite

It’s tempting to think a 1% or 2% flaky rate is fine. But small rates add up fast once you have lots of tests.

Tests in the Suite

Chance Each Test Passes

Chance the Whole Suite Goes Green

100

99.95%

About 95%

500

99.95%

About 78%

1,000

99.95%

About 61%

Even if each test only flakes once in 2,000 runs, a 1,000-test suite will fail for no real reason around four times out of ten. That’s why you can’t handle flaky tests one at a time in a big suite.

You need to track flakiness across the whole suite as a proper metric.



What Flaky Tests Really Cost You

The obvious cost is time spent rerunning and rechecking tests. The bigger costs are harder to see.

Cost

What Happens

People stop trusting tests

Engineers get into the habit of hitting “retry” first and only looking into a failure if it happens again. Real bugs slip through that way

Slower pipelines

Retries add time to every run, and across a large team committing several times a day, that can add up to many hours a month

Coverage looks better than it is

Flaky tests still count towards coverage, even though nobody trusts what they say

More bugs reach production

A real failure gets written off as “just flaky” and makes it into a release. One incident like that can cost more than a year of retries

Frustrated engineers

Automation engineers end up chasing random failures instead of building anything, which wears people down



The 7 Main Causes of Flaky Tests

Most flaky tests come down to one of seven causes, and each one needs a different fix. Timing comes up the most.

A 2014 study of flaky test fixes in open source projects found about 45% were caused by tests not waiting properly for async work to finish, so it’s the first thing to check.

Cause

What Goes Wrong

How to Fix It

Timing and async behaviour

The test assumes a page has loaded, an animation has finished or an API has replied, and sometimes it hasn’t. Fixed sleeps hide the problem for a while

Wait for something specific, like an element becoming clickable or a request finishing, instead of a set time

Fragile element locators

Tests find elements with deep CSS selectors or XPath, which break whenever the page structure changes, even if nothing a user would notice has changed

Find elements by their label, role or position on the page, and use tools that suggest fixes when the page changes

Test data problems

Tests rely on data from other tests or leftovers from earlier runs, so they fail when the data isn’t what they expect

Have each test create its own data, run against a known state and clean up after itself

Unstable environments

A slow database, a patchy network, a third-party timeout or an environment redeployed mid-run

Keep test environments close to production, separate concurrent runs, and handle outside failures cleanly

Tests running in a different order

Tests that share state clash when they run in parallel or in a different order

Make every test independent, so it can run in any order on any machine

Third-party services

An outside service has a bad day, and your test fails even though your app is fine

Mock outside services in unit and integration tests, and test against the real ones on a set schedule

Results that change by design

Random IDs, timestamps, unsorted lists or AI-generated text mean exact checks fail no matter how well the test is written

Check properties that stay true, like a valid ID format or a time within a range, instead of exact values

Here’s what the timing fix looks like in practice. This version waits a fixed time and hopes for the best:

driver.findElement(By.id("place-order")).click();
Thread.sleep(3000); // sometimes 3 seconds isn't enough
String status = driver.findElement(By.id("order-status")).getText();
driver.findElement(By.id("place-order")).click();
Thread.sleep(3000); // sometimes 3 seconds isn't enough
String status = driver.findElement(By.id("order-status")).getText();
driver.findElement(By.id("place-order")).click();
Thread.sleep(3000); // sometimes 3 seconds isn't enough
String status = driver.findElement(By.id("order-status")).getText();
driver.findElement(By.id("place-order")).click();
Thread.sleep(3000); // sometimes 3 seconds isn't enough
String status = driver.findElement(By.id("order-status")).getText();

This version waits for the thing it actually needs:

driver.findElement(By.id("place-order")).click();
String status = new WebDriverWait(driver, Duration.ofSeconds(15))
    .until(ExpectedConditions.visibilityOfElementLocated(By.id("order-status")))
    .getText();
driver.findElement(By.id("place-order")).click();
String status = new WebDriverWait(driver, Duration.ofSeconds(15))
    .until(ExpectedConditions.visibilityOfElementLocated(By.id("order-status")))
    .getText();
driver.findElement(By.id("place-order")).click();
String status = new WebDriverWait(driver, Duration.ofSeconds(15))
    .until(ExpectedConditions.visibilityOfElementLocated(By.id("order-status")))
    .getText();
driver.findElement(By.id("place-order")).click();
String status = new WebDriverWait(driver, Duration.ofSeconds(15))
    .until(ExpectedConditions.visibilityOfElementLocated(By.id("order-status")))
    .getText();

And here’s the fix for results that change by design. Instead of checking an exact order ID, check that it looks like a valid one:

// Flaky: the ID is different every run
assertEquals("ORD-88231", order.getId());

// Stable: check the format instead
assertTrue(order.getId().matches("ORD-\\d{5}"));
// Flaky: the ID is different every run
assertEquals("ORD-88231", order.getId());

// Stable: check the format instead
assertTrue(order.getId().matches("ORD-\\d{5}"));
// Flaky: the ID is different every run
assertEquals("ORD-88231", order.getId());

// Stable: check the format instead
assertTrue(order.getId().matches("ORD-\\d{5}"));
// Flaky: the ID is different every run
assertEquals("ORD-88231", order.getId());

// Stable: check the format instead
assertTrue(order.getId().matches("ORD-\\d{5}"));



Flaky Tests vs Tests That Always Fail

Both show up as a red build, but they’re very different problems.


Test That Always Fails

Flaky Test

Can you reproduce it?

Yes, every time

Not reliably

Is the result honest?

Yes, something really is wrong

No, it fails without a real reason

How hard is it to fix?

Usually straightforward

Hard, because you can’t make it fail on demand

What it does to the team

Points at a real problem

Teaches people to ignore red builds, including real ones

What to do

Fix it straight away

Track it, move it aside and investigate it properly



How to Find Flaky Tests

Flaky tests don’t label themselves, so you have to look at results over many runs to spot them.

Look at Pass and Fail History

A test that usually passes but occasionally fails, with no related code change, is a strong candidate. Agree on a threshold that makes sense for your team, rather than relying on a fixed industry number.

Track How Often Retries Pass

If your CI retries failed tests, a test that keeps passing on the second try is very likely flaky. Retries hide the problem, so log them and keep an eye on the numbers.

Check Where and When it Fails

A test that fails on one CI machine but not others, or only when the system is busy, is reacting to the environment, not the code.

Run it Lots of Times

Run a suspect test 50 or 100 times on the same build. If the results vary, it’s flaky.

Watch the Trend

One odd failure might be a one-off. The same pattern over weeks is flakiness. A chart of your flaky rate over time catches what single reports miss.



How to Manage Flaky Tests: A Six-Step Process

Finding flaky tests isn’t enough on its own. You need a clear process for dealing with each one.

Step

What Happens

Detect

Flaky tests get flagged automatically based on failure rate, retry results and where they fail, instead of engineers noticing the same retries over and over

Quarantine

The test moves out of the suite that blocks merges. It still runs and reports, but it can’t hold up a release. Quarantine is a waiting room, not a permanent home

Diagnose

Someone is given time to work out the cause. This is the most expensive step and the one teams skip most, which is how quarantine fills up

Fix or delete

Fix the test based on its cause, or delete it if it’s no longer worth keeping. A smaller suite you trust is better than a bigger one you don’t

Check the fix

Run the fixed test many times on the same build. Passing once doesn’t prove anything

Bring back or retire

Tests that stay stable go back into the main suite. Tests that don’t go back to diagnosis or get retired

Fewer flaky tests also makes it easier to decide which tests to run on each change. Our guide to risk-based regression testing covers how to choose which tests to run based on what changed.



Tools for Spotting Flaky Tests

Option

How It Helps

CI platform features and plugins

Several CI tools can flag flaky tests, either built in or through plugins. Azure DevOps and CircleCI have built-in flaky test reporting, and Jenkins has plugins for it. Check what your own CI tool offers

Dedicated flaky test tools

Track results across runs and spot patterns that simple retries miss. Many read JUnit, pytest and Jest results directly

Your own dashboards

Bigger teams often build their own, so they can link flakiness to deployments, environment changes and code owners

AI failure analysis

AI can look at failure patterns, timing and retry results to tell flaky failures apart from real but rare bugs



How to Stop Flaky Tests Before They Start

Detection deals with flaky tests after they show up. Prevention stops them showing up in the first place.

Wait for Things, Not for Time

Every wait should be for something specific in the app, not a fixed number of seconds. A Thread.sleep is a warning sign.

Describe What the User Does, Not How the Page is Built

Tests written around user actions survive redesigns much better than tests tied to today’s HTML.

Use Tools That Suggest Fixes When the Page Changes

When an element moves or gets renamed, a good tool finds the new one and proposes an update for someone to check, instead of failing or silently changing the test.

Look After Your Test Data

Fresh, versioned or synthetic data means tests always start from a known state.

Keep Tests Independent

Tests that can run in any order on any machine are safe to run in parallel.

Mock Unreliable Outside Services

Test your app separately from the services it depends on, and test the real services on a schedule.

Check Properties, Not Exact Values

Where results change by design, check the format, range or allowed values.

Independent, stable tests are also what make parallel runs safe. Our guide to parallel test execution explains how to run tests side by side without them tripping over each other.



Flaky Tests and AI-Written Code

AI now writes a lot of the code in many teams, and that makes flaky tests a bigger deal in three ways:

  • More code means more test runs: More pull requests and deployments mean tests run more often, so flaky failures happen more often too.

  • More refactoring means more broken locators: AI tools rewrite code regularly, changing page structures and IDs faster than many test suites can keep up with.

  • Less patience for noise: If you ship every day, you can’t afford a test suite your team has stopped believing.

None of this makes flaky tests acceptable. It means the habits that prevent them, like waiting for real events, describing user actions, looking after data and keeping tests independent, are now essential rather than nice to have.



How Virtuoso QA Cuts Down Locator-Driven Flaky Tests

Most causes of flaky tests need careful, case-by-case work. Broken locators are one of the biggest causes in UI test suites, and they’re also the one AI-written code makes worse, because every refactor shuffles the page structure.

That’s the cause Virtuoso QA is built to tackle.

  • Tests that point at what users see: A Virtuoso QA step like “click Submit order” finds the button by its label and surroundings, not a long CSS path. If a developer renames a class or wraps the button in a new div, the test doesn’t care.

  • A heads-up instead of a red build: If a page change does trip a step up, Virtuoso QA works out which element the step was after and suggests the update. Nothing changes in the test until someone on your team approves it, and the decision is logged, so a real bug can’t be quietly patched over as flakiness.

  • Built-in waiting: Each step waits for the page to be ready before acting, so you don’t end up with Thread.sleep calls scattered through your tests.

  • Same result every run: Tests run on a deterministic engine, so if the app and the test haven’t changed, the outcome shouldn’t either. When results do differ, that’s a strong sign something in the environment or data is off.

  • Clues when something fails: The failed step shows up next to the last passing run of the same step, with screenshots and logs, so it’s quicker to tell a flaky failure from a real one.


Stable tests are what make regression worth running on every change. To see how Virtuoso QA keeps regression suites dependable through constant releases, take a look at regression testing.



Related Reads

Flaky Tests FAQs

What is a flaky test?

What causes flaky tests?

How do you find flaky tests?

Should you just retry flaky tests?

What does quarantining a flaky test mean?

How do you stop flaky tests from happening?

Written by

Abhilash

Industry Analyst, Test Automation

Abhilash tracks how AI is reshaping software testing, from pairwise coverage to regulatory accountability, and writes research-led pieces that cut through vendor noise with numbers.

See what your next release looks like with Virtuoso

Book a walkthrough on your applications and your workflows. Bring a requirement, a user journey, or a brittle Selenium script, and watch the loop run on something you recognise.

See what your next release looks like with Virtuoso

Book a walkthrough on your applications and your workflows. Bring a requirement, a user journey, or a brittle Selenium script, and watch the loop run on something you recognise.

See what your next release looks like with Virtuoso

Book a walkthrough on your applications and your workflows. Bring a requirement, a user journey, or a brittle Selenium script, and watch the loop run on something you recognise.

Virtuoso QA is establishing the standard of proof for software releases. Its governed QA loop turns business requirements into tests for any browser-based application: AI proposes, a deterministic engine executes, a person approves what matters, and every decision leaves evidence.

Trust Center

AICPA

SOC

WAVE STRONG PERFORMER

@ Copyright 2026 SpotQA, Creators of Virtuoso QA

Virtuoso QA is establishing the standard of proof for software releases. Its governed QA loop turns business requirements into tests for any browser-based application: AI proposes, a deterministic engine executes, a person approves what matters, and every decision leaves evidence.

Trust Center

AICPA

SOC

WAVE STRONG PERFORMER

@ Copyright 2026 SpotQA, Creators of Virtuoso QA

Virtuoso QA is establishing the standard of proof for software releases. Its governed QA loop turns business requirements into tests for any browser-based application: AI proposes, a deterministic engine executes, a person approves what matters, and every decision leaves evidence.

Trust Center

AICPA

SOC

WAVE STRONG PERFORMER

@ Copyright 2026 SpotQA, Creators of Virtuoso QA