Experiment Plan

Origin & evolution

The idea that a test must be planned before it runs is old, but its modern formalisation comes from the replication crisis in psychology and biomedicine. Researchers noticed that statistically significant results kept failing to reproduce, and a chief culprit was p-hacking: trying many analyses, then reporting only the one that crossed the significance threshold. A close relative, HARKing, hypothesising after the results are known, dressed up post-hoc patterns as predictions.

Pre-registration is the corrective. You write the plan, including the analysis, and lodge it with a date stamp before collecting data. Brian Nosek and colleagues at the Center for Open Science built the Open Science Framework to host these plans, and argued the case directly in The preregistration revolution (PNAS, 2018). Their core claim is that pre-registration sharpens the line between hypothesis generation and hypothesis testing, so a reader can tell which results were predicted and which were discovered after the fact.

A useful debate followed. Alison Ledgerwood pointed out that a pre-registration mixes two separable things: the prediction and the analysis plan. Confirming a registered prediction with a registered analysis is the strong case; everything else is exploration, which is valuable and honest as long as it is labelled as such. Product experimentation inherited this discipline wholesale. An A/B test with a metric and a stop rule chosen up front resists the same biases that pre-registration was built to defeat.

Eric Ries made the same logic central to product practice in The Lean Startup, which frames a startup's core activity as running experiments🧫ExperimentValidationA test designed to validate a hypothesisView reference → rather than executing plans. His Build-Measure-Learn loop treats every product bet as a testable hypothesis: build the minimum version needed to generate data, measure a pre-chosen outcome, and learn whether the assumption🤔AssumptionStrategyA belief taken as true that underpins a strategyView reference → holds. The experiment plan is what keeps that loop honest — without a success criterion committed before build, the measurement step becomes an audit of whatever the data happens to show.

How it works in practice

A growth team believes a shorter signup form will lift completions. The experiment plan states the hypothesis precisely: cutting the form from nine fields to four raises signup completion from a baseline of 38% by at least three percentage points. It fixes the method (a 50/50 split on new visitors), the primary metric (completion rate), and a power calculation that says roughly 14,000 visitors per arm are needed to detect that effect. It names a stop rule: run for two full weeks, no peeking-and-stopping when the line looks good. It also pre-commits a guardrail metric, downstream activation, so a cheap completion win that produces worse users gets caught. With all of that written before launch, the result is interpretable whichever way it lands.

Experiment plan vs. its neighbours

• Hypothesis is the testable belief, a single falsifiable statement. The experiment plan is the apparatus built around it: how the hypothesis will be put at risk🎲RiskComplianceA risk to the product or businessView reference →. One hypothesis can demand several experiment plans before it is settled.
• Experiment run▶️Experiment RunValidationAn execution instance of an experiment that records actual conditions, observations, and raw results.View reference → is a single execution of the plan against real users at a real time. The plan is the design; the run is one instance of carrying it out. Separating them lets the same plan be run twice, on different cohorts👥CohortGrowthA group of users sharing a common characteristicView reference →, and compared honestly.
• Metric is what gets measured. The plan names a metric as its success criterion and points at it, but the metric exists independently and is usually tracked across many experiments.

In the graph

In the Unified Product Graph, experiment_plan📐Experiment PlanValidationAn experiment plan describing the hypothesis, setup, success criteria, and methodology before a test runs. sits in the validation region as the bridge between belief and evidence🔎EvidenceValidationData supporting or refuting a hypothesisView reference →. A hypothesis reaches for it through hypothesis_requires_experiment_planHypothesisrequiresExperiment Plancausal, the plan points at its yardstick through experiment_plan_targets_metricExperiment PlantargetsMetriccross-domain, and execution is recorded as a distinct node linked by experiment_plan_ran_as_experiment_runExperiment Planran asExperiment Runhierarchy. Growth work connects through growth_campaign_tests_via_experiment_planGrowth Campaigntests viaExperiment Planhierarchy. Keeping plan, run, and result as separate connected nodes is the structural version of pre-registration: the design is committed and queryable before any outcome is attached to it, so nobody can quietly rewrite the test to fit the answer.