Experiment Run

Origin & evolution

The discipline of running product experiments🧫ExperimentValidationA test designed to validate a hypothesisView reference → to settle questions comes from the lean and growth traditions. Eric Ries built *The Lean Startup* (2011) around the Build-Measure-Learn loop and the idea of validated learning🧠LearningValidationAn insight gained from an experimentView reference →, where a value hypothesis🧪HypothesisValidationA testable belief about a solutionView reference → and a growth hypothesis are tested against real customer behaviour, not asserted from the desk. A run is the "measure" leg of that loop made concrete: the loop turns once per run.

The statistical machinery is older and arrived with a warning attached. Online controlled experiments inherited the frequentist test, and with it the peeking problem. Evan Miller's widely-read How Not To Run an A/B Test showed that checking results repeatedly and stopping the moment significance appears inflates the false-positive rate far beyond the nominal five per cent. Ramesh Johari and colleagues formalised the fix in the 2017 KDD paper Peeking at A/B Tests, introducing always-valid p-values and sequential tests that stay honest under continuous monitoring. The lesson reshaped how a run is defined: a run has a pre-registered sample size or a sequential stopping rule, and analysing it is part of the run, not a separate liberty the analyst takes whenever the dashboard🎛️DashboardData & AnalyticsAn analytics dashboardView reference → looks good.

Where the field landed is a clean separation of three things. The plan states the hypothesis, the metric🔢MetricStrategyA unified metric that measures progress, health, or behaviour across the productView reference →, the minimum detectable effect, and the stopping rule. The run executes that plan once. The result is what the run yields, kept distinct so that a failed run still counts as evidence🔎EvidenceValidationData supporting or refuting a hypothesisView reference → and a single plan can be run more than once without anyone pretending the reruns were one test.

How it works in practice

A team believes a shorter signup form will lift completion. The experiment plan fixes the success metric at signup-completion rate, sets a minimum detectable effect of two percentage points, and a sample size of 40,000 visitors per arm to reach 80 per cent power. That is the design, and it does not move.

The run starts on the first of the month and splits live traffic 50/50. Eleven days in, the variant🆎VariantGrowthA variant in an A/B testView reference → is up 3.1 points and someone wants to call it. The pre-committed rule says wait for 40,000 per arm; with a sequential design, the always-valid boundary has not been crossed. They hold. By day eighteen the lift settles at 1.4 points, below the threshold that justified the work. The run produced a clear result, just not the hoped-for one, and because the stopping rule was set in the plan, nobody can argue the team peeked their way to a phantom win.

Experiment run vs. its neighbours

• Experiment plan is the reusable design: hypothesis, metric, stopping rule. The run is one execution of it. The same plan can produce several runs across quarters or markets, and conflating plan with run is how reruns get laundered into a single inflated result.
• Hypothesis is the testable belief the run is built to challenge. The run does not contain the belief; it subjects it to evidence. One hypothesis may need🫀NeedUserA user need, pain, desire, or constraintView reference → several runs before it is confidently validated or killed.
• Learning is the durable conclusion a run produces, the thing the team carries forward. A run yields evidence; the learning is the interpretation that survives. A run with no recorded learning has spent traffic and bought nothing.

In the graph

In the Unified Product Graph, experiment_run▶️Experiment RunValidationAn execution instance of an experiment that records actual conditions, observations, and raw results. sits in the validation region as the instance node between design and conclusion. experiment_plan_ran_as_experiment_runExperiment Planran asExperiment Runhierarchy records that a run is one execution of a plan, which makes reruns first-class and countable. experiment_run_validates_hypothesisExperiment RunvalidatesHypothesiscausal ties the run to the belief under test, experiment_run_yields_evidenceExperiment RunyieldsEvidencecausal captures the raw measured outcome🏆OutcomeStrategyA desired business or user outcomeView reference →, and experiment_run_produces_learningExperiment RunproducesLearningcausal records what the team concluded. Separating evidence from learning matters: it preserves the audit trail when a later run contradicts an earlier one, and it makes the peeking problem structurally visible, because a hypothesis "validated" by a single run with no pre-registered plan is queryably weak.