A commit lands on the trunk pipeline and one check goes red. The engineer looks at it, recognises the test, and says what everyone on the team says several times a day: “that one's just flaky, rerun it.” The rerun goes green, the commit moves on, and nobody writes anything down. Multiply that by every commit the team ships in a day and the flake has stopped being a test problem and become a tax on the whole delivery process. The test is annoying in isolation. Flaky tests in CI/CD (sometimes written CICD) sit directly on the path from commit to production, and that is what makes them expensive.
They cost more than the wasted minutes suggest, because the pipeline is a shared, gating, trust-dependent system and a flake damages all three properties at once. This is the CI/CD-context companion to the broader pipeline stability model: less about what makes a single test flaky, more about what flakiness does once it is running inside a pipeline every team depends on.
Why flaky tests in CI/CD are a pipeline problem, not just a test problem
A flaky test in a script you run locally is a nuisance you can ignore. The same test in CI/CD does three things a local flake never could. It gates the trunk the whole team builds on, so a false red halts integration for everyone, not just the engineer who pushed the change. It trains the team to distrust red, so the next genuine failure gets the same reflexive rerun and ships a real bug. And it degrades the one signal the pipeline exists to provide, a trustworthy yes or no on whether the code on trunk is safe to build on and release. Once “red might just mean flaky” is the working assumption, the pipeline has stopped being a gate and become a suggestion.
That is the CI/CD-specific damage. It is not that the test is unreliable; it is that an unreliable test wired into a gating system makes the gate unreliable, and a gate nobody trusts is worse than no gate at all, because it still costs time to clear.
Where flakiness surfaces in the pipeline
The mechanics are similar across GitHub Actions, GitLab CI, Bitbucket Pipelines, CircleCI, Azure DevOps, and Jenkins, because they all share the same shape: run the suite, record verdicts, expose a rerun control, and gate progress on the result. Three pipeline features quietly generate the evidence that a test is flaky, if anyone is reading it.
- Test reports.Almost every provider ingests JUnit-format XML, the de facto standard for machine-readable test results. Each run appends another verdict to a test's history, and that history is where flakiness is legible: a test that flips between pass and fail on the same commit is flaky by definition, because the code did not change between the runs.
- The rerun button. Every provider offers one, and it is the single most-used flakiness workaround there is. The problem is that a blind rerun cannot tell a flaky test from a broken one that happened to pass once, so used on its own it quietly reintroduces the exact conflation that makes flakiness dangerous.
- Matrix jobs and reruns. When the same commit runs across parallel matrix jobs, or is run again on an unchanged commit, and the verdicts disagree, that divergence is a flake caught in the act. The inputs were identical; only the result moved. It is a low-cost detection signal that needs no extra runs, only attention to results the pipeline already produced.
Why they cost more than you think
The visible cost is the rerun: the pipeline ran twice, so compute doubled. That is the small half. The larger half is wait time. Trunk-based teams integrate continuously, and the stop-the-line rule is that a red trunk halts feature work until it is green again. So a flaky failure does not cost only the engineer who hit it. It stalls everyone waiting on a green trunk to build on, and that wait multiplies across the team. A flake that adds ten minutes to one pipeline run adds ten minutes for everyone blocked behind a red trunk, and the cost scales with team size rather than with the number of flaky tests. This is why a handful of flaky tests on a busy repo can dominate the delivery cost while barely registering on the compute invoice. The full arithmetic, with the compute-doubled and wait-doubled breakdown, is worked through in the cost of flaky tests, which is the canonical cost treatment this section links across to rather than repeating.
There is a compounding effect on top of the arithmetic. The more a team reruns rather than fixes, the more flakes accumulate, because a rerun culture removes the pressure that would otherwise force a fix. Flakiness left alone does not stay flat; it grows, and it grows fastest on the busiest repos where the reruns are cheapest to reach for and the wait cost is highest.
Detection and fixing are separate jobs
Reading verdict history to identify which tests are flaky is a different task from repairing them, and conflating the two is why teams stall. Detection is a data problem: track flip rate, failure rate, and age of last green across runs, and the flaky tests sort themselves out from the healthy and the broken without adding any rerun load. The detection strategies cover how to gather those verdicts cheaply at scale. Fixing is a code problem: once a test is flagged and its root-cause class identified, the diagnostic checklist for fixing flaky tests picks up from there. Detection tells you which ten tests are causing most of the pain; fixing decides what to do about each.
How CI/CD Watch surfaces flakiness across providers
CI/CD Watch, a CI/CD observability platform that monitors pipelines across GitHub Actions, GitLab CI, Bitbucket Pipelines, CircleCI, Azure DevOps, and Jenkins, reads the evidence the pipeline already produces rather than adding runs to hunt for it. It parses the JUnit reports, tracks each test's verdict history, and computes flip rate, failure rate, and age of last green to classify every test as healthy, flaky, or broken, on one model across every connected provider rather than a different tool per pipeline. Because detection is history-based, it adds no reruns of its own, and because it is cross-provider, a team mid-migration between two providers sees one flakiness picture instead of two partial ones.
The output is a ranked list: which tests are flaky, ordered by how often they fire, so the pipeline's worst offenders are already prioritised. The classification is the signal; whether to repair, quarantine, or delete each test stays a human decision. The Free tier covers pipeline-run monitoring across all six providers; the per-test flakiness detail sits on the paid tiers.
Stop paying the flake tax
If your team reruns red checks on reflex and nobody can name the tests responsible, connect your providers and CI/CD Watch classifies each test from its verdict history and ranks the flaky ones by impact. Monitoring is free; the per-test detail is on the paid tiers. For how the detection actually works against the JUnit reports your pipelines emit, the flaky-tests documentation covers the mechanics, and the pipeline stability overview sets out the healthy, flaky, and broken model these signals feed.
CI/CD Watch is built by 3CS Technologies Ltd. It started as an internal tool for tracking pipeline health across a mixed GitHub Actions and Jenkins estate. The same engine now powers the SaaS platform.