Test Result

Origin & evolution

The pass/fail verdict is foundational to automated testing, and for most of its history the model was binary: a test asserted something, and the assertion either held or it did not. Skip arrived as an honest third state for tests deliberately not run, on an unsupported platform, behind a disabled feature flag🎌Feature FlagEngineeringA feature toggle for controlled rolloutView reference →, or pending a fix.

Kent Beck's formulation of test-driven development made the pass/fail verdict a tight feedback loop: write a failing test, write the minimum code to make it pass, then refactor. In that cycle the result is not merely a report but the primary signal driving the next action, which is why the binary meaning of red and green matters so much — ambiguity in the verdict breaks the loop.

The fourth state, flaky, is the one that reshaped how teams read results. A flaky test produces both passing and failing outcomes🏆OutcomeStrategyA desired business or user outcomeView reference → against the same code, so its result carries no information about whether the product works. Google quantified the scale of the problem in its widely cited post, Flaky Tests at Google and How We Mitigate Them (2016): about 1.5% of all test runs were flaky, and roughly 84% of observed pass-to-fail transitions were caused by flakiness rather than a real regression. At that scale, a single run's result becomes nearly meaningless as a gate, because most red is noise.

That finding pushed the field from reading single results toward reading trends. Google built tooling to automatically quarantine tests above a flakiness threshold, removing them from the critical path and filing bugs🐛BugProduct SpecificationA defect or unexpected behaviourView reference → against them. The lesson generalised: a result is a data point, and the honest reading of a suite is the pattern across runs, not the verdict of the latest one.

Humble and Farley's deployment📤DeploymentEngineeringA deployment eventView reference → pipeline model formalised results as stage gates: each pipeline stage🧽Pipeline StageSales & RevenueA stage in the sales pipelineView reference → runs a class of tests and its aggregate result must be green before work advances to the next stage. On that model a result is not just a verdict on a single run but a condition that controls whether software can move toward production — which is why a persistently unreliable result damages more than a single build; it erodes the pipeline's authority as a release gate.

How it works in practice

A team's CI shows a failing build. The failed test is an end-to-end checkout test asserting that an order confirmation appears within three seconds. An engineer re-runs it; it passes. Re-runs again; fails. Same commit each time. The result is flaky, and the cause is a race between the test and an asynchronous email-send the page waits on.

Treating the single red as a real failure would have blocked a clean release for nothing. Reading the trend instead, the team sees this test has flipped on 9 of the last 40 runs with no related code changes. They quarantine it out of the merge gate, file a bug, and fix the race by waiting on an explicit confirmation event rather than a timeout. The result becomes deterministic again, and only then does it return to gating releases. The verdict regained its authority by becoming repeatable.

Test result vs. its neighbours

• Test case is what runs; the result is what running it produces. The case is the durable assertion you maintain; the result is the per-run, disposable verdict it emits. One case yields a new result on every execution, through test_case_produces_test_resultTest CaseproducesTest Resultcausal.
• Test suite📂Test SuiteQuality AssuranceA suite of related testsView reference → produces an aggregate result; the case produces a granular one. The suite-level result is the roll-up that gates the build through test_suite_produces_test_resultTest SuiteproducesTest Resultcausal; a single case result tells you which assertion moved.
• Bug is a confirmed defect; a failing result is a candidate for one. A red result might be a real regression, a flaky test, or a wrong assertion. The triage step is deciding which, and only the first becomes a bug.

Michael Feathers introduces a third reading of a result that is relevant here: the characterization test. Where a normal test asserts what code should do, a characterization test records what code currently does, and its result documents existing behavior rather than desired behavior. A red result from a characterization test does not mean the code is wrong; it means behavior has changed. That distinction — whether a result is checking a contract or pinning an observation👁️ObservationUser ResearchA specific behaviour or statement observedView reference → — shapes how a team should act on a failure.

In the graph

In the Unified Product Graph, test_result🎰Test ResultQuality AssuranceA test execution result sits in the quality and testing region as the emitted signal of verification. It is produced both at the granular level through test_case_produces_test_resultTest CaseproducesTest Resultcausal and at the aggregate level through test_suite_produces_test_resultTest SuiteproducesTest Resultcausal. Modelling the result as its own node, separate from the case that emits it, is what lets the graph hold history: a sequence of results over time exposes flakiness, surfaces trends, and turns "the build is red" into the more useful question of whether this red has been red before.