Ask a team about their CI pipeline stability and you usually get a shrug and a qualifier. “Mostly?” “It was bad last week.” “The integration suite is a nightmare but the rest is fine.” Everyone has a feeling and nobody has a number. The pipeline goes red some mornings and green others, people rerun it, and whether that is getting better or worse over the quarter is anyone's guess. A dashboard pass rate of, say, 82% tells you almost nothing on its own, because it lumps a genuinely broken job in with a suite that flakes once a day and a workflow that is perfectly healthy.
CI pipeline stability is not a single percentage. It is a classification: every test and every pipeline is either healthy, flaky, or broken, and the whole point is to tell those three apart rather than average them into one misleading number. This is the measurement companion to the broader pipeline stability model, focused on the specific signals and thresholds that draw the lines between the three categories.
Healthy, flaky, broken: the only three states that matter
The same three categories apply at the test level and at the pipeline level. A healthy test passes consistently across recent runs. A flaky test produces mixed pass and fail verdicts with no clear cause in the code that changed. A broken test fails consistently, or has no green run left in the recent window. The distinction is not academic, because the response to each is different. A healthy test needs nothing. A flaky test needs triage, a fix, or a quarantine. A broken test needs the test or the code behind it fixed. Collapse flaky and broken into one “just rerun it” bucket, as most teams do under pressure, and you rerun genuine failures until they mask a real bug while you also stop trusting the tests that were telling the truth.
At the pipeline level the same labels aggregate up. A pipeline is flaky if its jobs flip verdicts without tracking the code changes, broken if its failure rate is sustained, and healthy otherwise. Classifying at both layers matters because it tells you where the fix lives. A pipeline can be flaky because one specific test is flaky, in which case the fix is at the test. Or it can be flaky because of infrastructure shared by many tests, a flaky runner image or an overloaded shared service, in which case no single test fix will help. The pipeline-level view tells you which problem you actually have.
The three signals that classify a test
For an individual test, three signals read together draw the line between the categories. Read alone, each one misleads.
- Flip rate. The proportion of consecutive run pairs where the test changes verdict on the same code. Above 5% suggests flakiness, and the higher the number, the more confident the call. Flip rate captures instability: how often the verdict moves when the code did not.
- Failure rate over the recent window. How much of the time the test is red. Used with flip rate, it separates the two failing states. For example, a test failing 90% of recent runs with a low flip rate is broken; one failing 30% with a high flip rate is flaky.
- Age of last green.A test that has not passed in two weeks is probably broken regardless of flip rate. Time- bounding the window stops old reds being classified forever as “just flaky” and quietly ignored.
The reason all three are needed is that any one of them has a failure mode. A rarely-run test that flakes has a high flip rate and a low failure rate. A consistently-failing test has a low flip rate and a high failure rate. Age of last green catches the test that is slowly trending from flaky to broken while its flip rate still looks alive. Together they classify; separately they argue.
Aggregating to the pipeline
The same logic scales up to the pipeline, with thresholds tuned for the noisier aggregate signal. A useful working rubric: mark a pipeline broken when its recent failure rate crosses 80%, flaky when its flip rate crosses 30% and at least one failure has been observed, and healthy otherwise. Treat the flaky pipelines that flip verdict on more than half their consecutive run pairs as high severity, because those are the ones eroding trust fastest.These are the thresholds the CI/CD Watch classifier uses, and they are documented in full in the pipeline-stability docs.
The exact numbers matter less than the discipline of having them. A team that agrees “broken means no green in two weeks” and “flaky means flipping more than three times in ten” can act on the classification without re-litigating it every standup. A team that argues each red on its merits burns the same argument daily and never builds a backlog. The thresholds are a shared definition, not a physical constant, so tune them to your suite and then hold them steady.
CI stability metrics worth tracking over time
A single snapshot classifies; a trend tells you whether the practice is improving. The stability metrics worth watching are the aggregates of the per-test signals: the count of pipelines in each category week over week, the flip rate of the pipelines on your critical path, and the age of the oldest un-green test blocking a merge. If the flaky count is climbing, the suite is drifting and rerun culture is filling the gap. If the broken count is climbing, something is decaying and being ignored. This is exactly the kind of cross-run pattern a red or green dashboard cannot surface, which is why stability belongs in the broader set of signals worth monitoring in CI/CD rather than in a standalone tool. The mechanics of turning raw verdicts into these numbers get their own treatment in the provider-specific view of flakiness.
How CI/CD Watch measures CI pipeline stability
CI/CD Watch, a CI/CD observability platform that monitors pipelines across GitHub Actions, GitLab CI, Bitbucket Pipelines, CircleCI, Azure DevOps, and Jenkins, applies exactly this rubric automatically. It parses the JUnit test reports the providers already produce, tracks each test's verdict history, and computes flip rate, failure rate, and age of last green to classify every test and every pipeline as healthy, flaky, or broken. Detection is history-based, so it adds no reruns of its own, and it runs the same classifier across every connected provider rather than reinventing it per tool.
The trend view is where the classification earns its keep: category counts over time turn “the suite feels worse lately” into a line you can point at in a retro. The classification is the signal; deciding whether to repair, quarantine, or delete a flaky test is still a human call. The Free tier covers pipeline-run monitoring across all six providers; the per-test and per-pipeline stability detail sits on the paid tiers.
Put a number on your pipeline stability
If the honest answer to “is our pipeline stable?” is a shrug, connect your providers and let CI/CD Watch classify every pipeline as healthy, flaky, or broken from its run history. The Free tier covers pipeline monitoring; the stability detail is on the paid tiers. For the full model behind the classification, including the six root causes of flakiness and how rerun culture hides them, the pipeline stability overview covers it end to end.
CI/CD Watch is built by 3CS Technologies Ltd. It started as an internal tool for tracking pipeline health across a mixed GitHub Actions and Jenkins estate. The same engine now powers the SaaS platform.