Evaluation Run

Origin & evolution

The run-as-record discipline borrows directly from software testing. A test suite📂Test SuiteQuality AssuranceA suite of related testsView reference → is the fixed thing; a test run is one invocation against one build, and continuous-integration culture taught teams to store every run so they can see the moment a green suite turned red. Applied to models, the same idea produces the offline eval: a suite of inputs with expected behaviour, executed before anything ships.

The hard part is scoring open-ended generations, where there is no single correct string. The answer that scaled was LLM-as-a-judge, formalised by Lianmin Zheng and colleagues in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023). They showed that a strong judge model such as GPT-4 agrees with human preferences over 80% of the time, matching the agreement rate between humans themselves. That made automated, repeatable eval runs practical for subjective quality, and the same paper named the position and verbosity biases that a careless run inherits from its judge.

Practice then folded eval runs into the deployment📤DeploymentEngineeringA deployment eventView reference → pipeline. The widely repeated rule in modern AI engineering is that prompts are code, which makes a regression eval the natural gate: run the suite on every prompt or model change, block the change if the score drops. The run stops being a one-off experiment🧫ExperimentValidationA test designed to validate a hypothesisView reference → and becomes a recurring check, like a unit-test job🔨JobUserJob To Be Done: what the user is trying to accomplishView reference →.

How it works in practice

A team ships a summarisation feature⭐FeatureProduct SpecificationA product capability or featureView reference → with a 120-example offline eval, each example paired with a reference summary and a judge rubric scoring faithfulness and concision. The baseline eval run on prompt version 4 scores 8.1 out of 10. An engineer rewrites the system prompt to cut verbosity and opens a pull request. The CI regression eval fires automatically: prompt version 5 scores 8.4 on concision but 7.2 on faithfulness, because the tighter prompt started dropping caveats. Two stored runs, side by side, make the trade-off legible, and the change is held until the faithfulness drop is fixed. Without the recorded baseline run, the regression would have reached users as a quiet quality decline.

Eval run vs. its neighbours

• Eval benchmark🎖️Eval BenchmarkAI & Machine LearningA benchmark for evaluating AI qualityView reference → is the reusable instrument: the dataset plus the scoring rule, held constant. The run is one execution of that instrument. You author a benchmark once and accumulate many runs against it.
• Metric🔢MetricStrategyA unified metric that measures progress, health, or behaviour across the productView reference → is the number a run produces. A run is the event of measuring; the metric is the measurement. One run can emit several metrics at once, which is why they are separate entities.
• AI experiment👾AI ExperimentAI & Machine LearningAn AI-focused experimentView reference → is a structured investigation🔍InvestigationEngineeringAn investigation into an issue or incidentView reference → that may compare several runs to test a hypothesis🧪HypothesisValidationA testable belief about a solutionView reference →. A run is the atomic execution an experiment is built from; the experiment is the question, the runs are the readings.

In the graph

In the Unified Product Graph, eval_run🎟️Eval RunAI & Machine LearningAn evaluation run against a benchmark lives in the ai domain alongside ai_model🤖AI ModelAI & Machine LearningAn AI or ML model used within the productView reference → and eval_benchmark🎖️Eval BenchmarkAI & Machine LearningA benchmark for evaluating AI qualityView reference →. Its defining edge is eval_benchmark_executed_as_eval_runEval Benchmarkexecuted asEval Runhierarchy, which ties every run back to the fixed benchmark that produced it, and eval_run_produces_metricEval RunproducesMetriccausal, which records the scores that run emitted. Modelling the run as its own entity, distinct from the benchmark, is what lets the graph hold a benchmark's full history: every execution, against every model build, over time. That history is the raw material for the Model Evaluation Loop, where models are compared across runs rather than on a single snapshot.