Experiment

Origin & evolution

Controlled experimentation is centuries old in science, but its move into product practice came through The Lean Startup (2011). Eric Ries argued that a startup is an engine for learning🧠LearningValidationAn insight gained from an experimentView reference →, and that experiments, not opinions or executive intuition, are how it earns validated knowledge. He paired this with innovation accounting: run an experiment, measure the effect on the metrics🔢MetricStrategyA unified metric that measures progress, health, or behaviour across the productView reference → that matter, and decide whether to persevere or pivot. The build-measure-learn loop made the experiment the heartbeat of the method.

The practice matured from "run an experiment" into "choose the right experiment". David Bland and Alexander Osterwalder's Testing Business Ideas (2019) catalogued 44 named experiments, sequenced from cheap and weak (interviews, landing pages, Wizard of Oz tests) to expensive and strong (working prototypes📱PrototypeExperience DesignAn interactive mockup for testingView reference →, paid pilots). Their rule of thumb reorders intuition: test desirability before feasibility, run the cheapest experiment that produces meaningful evidence, and aim it at the riskiest assumption🤔AssumptionStrategyA belief taken as true that underpins a strategyView reference → first. A polished test of a safe belief is theatre.

The cost-of-evidence frame is now the discipline's centre of gravity. Each experiment is judged by how much risk it removes per unit of time and money spent, which is why a fake door or a concierge test can beat months of engineering when the open question is whether anyone wants the thing at all.

How it works in practice

A fintech team believes small businesses will pay for automated invoice💳InvoiceSales & RevenueAn invoice for billingView reference → chasing. The riskiest assumption is desirability: will anyone actually pay? Building the feature⭐FeatureProduct SpecificationA product capability or featureView reference → would take a quarter, so they design the cheapest test that still draws real evidence. They ship a "smoke test" landing page describing the service with a "Start free trial" button, drive 2,000 visitors through a small ad spend, and define the decision rule in advance: a click-through to the pricing step above 8% justifies building a concierge version, anything under 4% kills the idea.

The result comes back at 11%. That is not proof the product will succeed, and they do not treat it as such. It clears one specific hurdle, desirability, cheaply, and licenses the next, more expensive experiment: a manually-operated concierge service for ten real customers to test whether they stay once the chasing actually happens. The experiments are staged so each one buys the right to run the next.

Experiment vs. its neighbours

• A/B test is one experiment type: a live, randomised, controlled comparison of two variants🆎VariantGrowthA variant in an A/B testView reference → on real traffic, strong for optimising an existing flow. The broader category of experiment includes tests with no live product at all, such as interviews and landing pages, used precisely when there is nothing yet to split-test.
• Prototype is an artefact you build to provoke a reaction; an experiment is the test that surrounds it, including the hypothesis, the metric, and the decision rule. A prototype with no measurement plan is a demo, not a test.
• Hypothesis is the prediction; the experiment is the procedure that judges it. A common failure is running activity (shipping, demoing, surveying) with no hypothesis attached, which generates data nobody can act on.

In the graph

In the Unified Product Graph, experiment🧫ExperimentValidationA test designed to validate a hypothesis is a self-nesting hub in the Discovery, Research & Validation region, sitting downstream of the hypothesis it tests. The structure separates intent from execution: an experiment🧫ExperimentValidationA test designed to validate a hypothesis holds its design via experiment_has_planExperimenthas planExperiment Planhierarchy and its execution via experiment_executed_as_experiment_runExperimentexecuted asExperiment Runhierarchy, so a planned test and an actual run are distinct nodes. Output is first-class: the run validates the claim through experiment_run_validates_hypothesisExperiment RunvalidatesHypothesiscausal and measures its effect through experiment_run_measures_metricExperiment RunmeasuresMetriccross-domain, while the experiment yields experiment_produces_learningExperimentproducesLearningcausal and experiment_produces_evidenceExperimentproducesEvidencecausal. Because evidence and learning are their own entities, the graph can answer which beliefs have actually been tested, and which still rest on intuition.