Your Green CI Check May Have Run Zero Steps
A passing check is a claim about execution, not proof of it. Four distinct failure modes in one pipeline turned absence-of-failure into a success signal.
Evidence status: established. Job-level figures were read from the CI provider’s API during an outage that was in progress when this was first written up, in August 2026.
Scope. One pipeline, four failure modes. The reporting lesson is general; the outage cause is infrastructure-specific.
Next test. Add a post-run assertion that every required job recorded a non-zero step count, and fail the pipeline when it did not.
A CI result is a claim about execution. Everyone reads it as proof of execution. The gap between those two things is where a surprising number of defects live, and we hit four separate instances of it in one pipeline over a few weeks.
The most striking was the one in progress when we first wrote this up in August: every job in
our pipeline was reporting failure after 1 to 3 seconds, having executed zero steps, with
no runner assigned.
1. The August outage: seven jobs, zero steps#
| Job | Conclusion | Steps recorded | Duration | Runner |
|---|---|---|---|---|
unit | failure | 0 | 2s | (none) |
security | failure | 0 | 3s | (none) |
type-check | failure | 0 | 3s | (none) |
implement-smoke | failure | 0 | 3s | (none) |
egress-enforcement-rootless | failure | 0 | 1s | (none) |
cross-repo-integration | failure | 0 | 2s | (none) |
docker-e2e | failure | 0 | 2s | (none) |
Source dataset: ./data/ci-job-evidence.csv.
Scope of the outage:
| Metric | Value |
|---|---|
| Last successful run | 2026-08-09 18:24 |
| First run of continuous failure | 2026-08-09 18:53 |
| Runs sampled | 60 |
| Failed in sample | 52 |
| Distinct branches affected | 14 |
| Actions enabled | true |
Source dataset: ./data/outage-window.csv.
The diagnostic value is in the combination of signals, and the empty runner name is the one that resolves it.
A job that fails in 2 seconds having run no steps did not fail because of your code. It never
got a machine. runner_name is empty because no runner was ever assigned. This is a
runner-allocation or quota condition, not a code condition — and critically, Actions is
enabled, which rules out the first thing everyone checks.
That distinction matters because the two states look identical in the pipeline summary view.
A disabled-Actions repository and a repository with no available runners both produce a wall
of red. If you check the enabled flag, see true, and go hunting in your test suite, you can
lose a day. The tell is the triple: zero steps + empty runner + sub-5-second duration.
Note also that this failure is at least loud. It presents as red. The four failure modes below are the dangerous direction — they presented as green.
2. Gates that never ran because an earlier job failed#
A hard-fail quality gate — the kind that must pass before anything ships — was configured to run after the unit tests. When the unit tests failed, the gate did not run.
Not “ran and was skipped.” Not “ran and reported inconclusive.” It did not execute, and its absence rendered as an absence of failure.
The result: the gate that exists precisely to catch a bad state was structurally unreachable in exactly the situation that produces a bad state. It could only ever run when everything was already fine.
This is vacuous success: a rule that finds nothing may simply have matched nothing. A skipped job and a passing job occupy the same visual space in most CI interfaces — a non-red mark — and a reader scanning for problems finds none.
3. A ratchet that parsed nothing and reported clean#
We run ratchets: gates that read a report, compare it against a committed baseline, and fail if the numbers got worse.
One of them was parsing nothing. The report format had drifted, the parser matched zero entries, and zero entries compared favourably against the baseline. It reported clean.
Separately, the same gate treated an unreadable report as a pass. If the file was missing or malformed, the comparison had nothing to object to.
Both are the same error in different clothes: the failure of the measurement was interpreted as a measurement of no failures. A parser that extracts zero items from a non-empty report is broken, and it is trivially detectable — assert that the parse produced at least one entry when the source file is non-empty — but only if you think to check the instrument rather than the reading.
4. Credentials that silently masked themselves#
A deploy step with missing credentials was falling back to a cached artifact instead of failing. The pipeline went green. The artifact was stale.
The fix was to hard-fail on missing credentials. The general principle: a fallback path that produces a plausible result on a configuration error is a silent-failure generator. Caching is meant to be a performance optimization; here it had become an availability mechanism papering over a misconfiguration, and the pipeline could not tell the difference between “fetched successfully” and “could not fetch, used what I had.”
5. And six modules that were never in the pipeline at all#
The fourth mode is the quietest. Six unit-test modules do not run in CI. Not skipped, not failing — never collected. They exist in the repository, they pass locally, and the pipeline has never executed them.
There is no signal for this anywhere. A test that does not run produces no output, no warning, and no mark in the interface. The only way to detect it is to compare the set of tests that exist against the set that reported, which nothing does by default.
6. Do not trust the summary view; read the API#
Across all four modes, the pipeline’s own summary display was inadequate or actively misleading. Two specific lessons:
The watch command lied in both directions. Using the CLI’s run watch --exit-status to
gate on a result gave wrong answers both ways — reporting failure for runs that succeeded and
success for runs that did not. We stopped using it and now read conclusion from the API
directly.
Job-level and step-level evidence is where the truth is. A run-level conclusion collapses seven jobs into one word. The step count, the runner name, and the duration — the three fields that identified the August outage in under a minute — are not in the summary view at all.
The check we should have had from the start, and now recommend:
After the pipeline completes, assert that every required job recorded a non-zero step count and a non-empty runner. Fail if not.
This single assertion catches the August outage and the never-ran gate. It does not catch the six uncollected modules: those jobs ran steps, just not those tests, so they need the exists-versus-reported comparison from section 5. Nor does it catch the ratchet parsing nothing or the credential fallback — those are instrument-level checks that belong inside their own tools.
7. Counterfactuals#
Isn’t the August outage just an infrastructure incident, not a lesson? The outage is
infrastructure. The lesson is that it took explicit job-level inspection to distinguish “no
runners” from “our code broke”, and that a reasonable engineer checking the obvious flag
would have been misled by enabled: true.
Wouldn’t required-status-checks have prevented merges during all this? For the red failures, yes. For the green ones — the skipped gate, the vacuous ratchet, the credential fallback, the uncollected modules — required checks are actively harmful, because they attach the authority of a required gate to a signal with nothing behind it.
Is “zero steps” a reliable indicator? In our provider it is, but it is a heuristic, not a contract. A job that fails during setup can also record zero steps for legitimate reasons. The combination with an empty runner name is what makes it diagnostic here, and that combination is provider-specific.
Could all four have been caught by one policy? No, and we want to be clear about that. One, the gate that never ran, is a pipeline-topology defect catchable by the step-count assertion. One, the uncollected modules, needs a comparison of the tests that exist with the tests that reported. Two are instrument-level defects inside individual tools. The unifying idea — absence of failure is not evidence of success — is a mindset rather than a check.
8. Limitations#
- One pipeline, one provider. Field names and the zero-step heuristic will not transfer directly.
- The outage was still ongoing when this was first written, so its root cause is inferred from the signals (no runner assigned, Actions enabled) rather than confirmed by the provider.
- The 60-run sample is a recency window, not the full history; the outage may have intermittent green runs outside it.
- We have not implemented the step-count assertion. It is a recommendation derived from the incidents, not a measured intervention.
9. Takeaways#
- Zero steps + empty runner + sub-5-second duration means you never got a machine. Do not debug your code. Check quota and runner availability — even when Actions reports enabled.
- A job that did not run looks exactly like a job that passed. Assert that required jobs recorded steps.
- Check the instrument, not just the reading. A ratchet that parses zero entries from a non-empty report is broken and reports clean; a gate that treats an unreadable file as a pass is worse.
- Fallbacks on configuration errors manufacture false green. Missing credentials should fail, not silently serve a cached artifact.
- Read conclusions from the API, not from a watch command. Ours was wrong in both directions, which is the worst kind of wrong — you cannot even apply a consistent correction.
- Compare the tests that exist to the tests that reported. Six of ours had never run in CI, and nothing anywhere would have said so.