A codebase with ten thousand tests goes red intermittently several times a day. Everyone knows some of those failures are flaky and some are real, and nobody can say which tests are the culprits. The team has a rerun button and a shared suspicion, but no list. Asked “which tests are actually flaky?” the honest answer is that detection was never done systematically; it was outsourced to whoever happened to be watching the pipeline that morning.
Flaky test detection at scale is not rerunning everything and seeing what changes. It is reading the verdict history you already have. The reliable signal is how often a test flips between pass and fail on the same code, not whether a single retry happened to go green, and that signal classifies each test as healthy, flaky, or broken without adding any rerun load. This is the detection-mechanics companion to the pipeline stability model, which sets out the healthy / flaky / broken classification this post detects against.
Detection is a classification problem
Before a strategy, a definition. A flaky test produces different results on identical code and inputs across consecutive runs. A broken test fails consistently, with no green run in the recent window. A healthy test passes reliably. Detecting flaky tests means telling those three apart, and the trap most teams fall into is conflating flaky and broken under the single label “flaky, just rerun it.” Rerunning a genuinely broken test wastes compute and hides a real failure; quarantining a healthy test that failed once throws away a working signal. A detection strategy earns its keep by putting each test in the right one of the three buckets, not by flagging everything that ever went red.
Flaky test detection strategies, cheapest to most invasive
Rerun on failure and compare
The default almost every team reaches for: when a test fails, run it again; if it passes on retry, mark it flaky. It is trivial to switch on and it catches the obvious cases. The limits show up fast. It only sees tests that fail on the run you are watching, so a test that flakes once a fortnight slips through. It adds rerun load precisely when the pipeline is already slow. And a blind retry cannot separate a flaky test from a broken one that happened to pass once, so used alone it quietly reintroduces the conflation the classification is meant to prevent.
Verdict-history analysis (flip rate)
The strongest signal is already sitting in your run history. Track each test's sequence of pass and fail verdicts across runs on the same code, and the flip rate, the proportion of consecutive run pairs where the verdict changed, separates flaky from broken directly. A test that fails ninety per cent of recent runs with a low flip rate is broken; one that fails thirty per cent with a high flip rate is flaky. This is the strategy that scales, because it reads data every provider already produces and adds no runs of its own. For a large codebase, history-based detection is the primary mechanism and rerun-based detection is the fallback, not the other way round.
Deliberate repeated runs
To confirm a specific suspect, run it many times against an unchanged commit and watch for a mixed result. This forces a flake that history only hints at, and it gives the highest-confidence answer. It is also the most expensive strategy, so it belongs in triage of individual suspects surfaced by history analysis, not as a fleet-wide sweep. Running ten thousand tests a hundred times each to find the flaky ones is a way to detect flakes and manufacture a compute bill at the same time.
Same-commit divergence
When the same commit produces different verdicts, across parallel matrix jobs, across reruns, or across a merge queue re-execution, that divergence is a flake by definition, because the code did not change between them. Watching for verdict disagreement on identical inputs is a low-cost detection signal that needs no extra runs, only attention to results you were already generating.
What changes at large-codebase scale
At ten tests, you can rerun everything and eyeball the results. At ten thousand, the strategy has to change shape. You cannot afford rerun-based detection as the primary mechanism, so history-based detection wins on cost alone. Detection also stops being a yes/no per test and becomes a ranked list: thousands of tests, each classified, ordered by how much they actually cost you. A flaky test in a rarely-run suite matters less than one sitting on the critical path of every pull request. Detecting flakes is only half the job at scale; the other half is prioritising which ones to detect loudly, so the team fixes the ten tests causing eighty per cent of the reruns rather than the long tail that fires once a month. That prioritisation is where the cost of flaky tests turns a detection list into a work queue.
The three signals that classify a test
Whatever strategy feeds the data, three signals read together do the actual classification. Flip rate captures instability: how often the verdict changes on the same code. Failure rate over the recent window captures severity: how much of the time the test is red. Age of last green captures direction: a test with no green run in weeks is trending broken, not flaky. Read alone, each misleads. A test failing intermittently but rarely has a high flip rate and a low failure rate, and it is genuinely flaky. A test failing almost always has a low flip rate and a high failure rate, and it is broken. The same rubric aggregates from test level up to job and pipeline level, which is how a whole pipeline gets classified rather than just individual tests. Detection strategies differ in how they gather verdicts; the classification rubric is what turns those verdicts into an answer. The fix patterns for flaky tests pick up once a test is classified and its root-cause class identified.
How CI/CD Watch detects flaky tests
CI/CD Watch, a CI/CD observability platform that monitors pipelines across GitHub Actions, GitLab CI, Bitbucket Pipelines, CircleCI, Azure DevOps, and Jenkins, treats flaky tests as a first-class signal. It parses the JUnit test reports the providers already produce, tracks each test's verdict history, and computes flip rate, failure rate, and age of last green to classify every test as healthy, flaky, or broken. Detection is history-based, so it adds no reruns of its own, and it works across every connected provider on one model rather than per-provider.
The output is the ranked list scale demands: which tests are flaky, sorted by how often they fire, so the detection result is already a prioritised work queue rather than an undifferentiated pile. Detection is the signal; the fix is still a human decision about whether to repair, quarantine, or delete each test. The Free tier covers pipeline-run monitoring across all six supported providers; the per-test stability detail sits on the paid tiers.
Detect flaky tests across your suite
If your CI goes red intermittently and nobody can name the tests responsible, connect your providers and CI/CD Watch classifies each test as healthy, flaky, or broken from its verdict history and ranks the flaky ones by impact. The Free tier covers pipeline monitoring; the per-test detection detail is on the paid tiers. For the classification model these signals feed, the pipeline stability overview covers the healthy / flaky / broken rubric and the six root causes in full.
CI/CD Watch is built by 3CS Technologies Ltd. It started as an internal tool for tracking pipeline health across a mixed GitHub Actions and Jenkins estate. The same engine now powers the SaaS platform.