Evaluation Run

Origin & evolution

The run-as-record discipline borrows directly from software testing. A test suite📂Test SuiteQuality AssuranceA suite of related testsView reference → is the fixed thing; a test run is one invocation against one build, and continuous-integration culture taught teams to store every run so they can see the moment a green suite turned red. Applied to models, the same idea produces the offline eval: a suite of inputs with expected behaviour, executed before anything ships.

The hard part is scoring open-ended generations, where there is no single correct string. The answer that scaled was LLM-as-a-judge, formalised by Lianmin Zheng and colleagues in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023). They showed that a strong judge model such as GPT-4 agrees with human preferences over 80% of the time, matching the agreement rate between humans themselves. That made automated, repeatable eval runs practical for subjective quality, and the same paper named the position and verbosity biases that a careless run inherits from its judge.

Chip Huyen's *Designing Machine Learning🧠LearningValidationAn insight gained from an experimentView reference → Systems* (O'Reilly, 2022) codifies what a run record should contain and why the discipline matters: she distinguishes experiment🧫ExperimentValidationA test designed to validate a hypothesisView reference → *tracking* — capturing loss curves, per-split metrics🔢MetricStrategyA unified metric that measures progress, health, or behaviour across the productView reference →, inference speed, and sample-level predictions — from experiment *versioning*, which logs the configuration parameters that would be needed to reproduce the run. By that framing, an eval run is the intersection of both: a timestamped execution whose score is only meaningful if the model build and prompt config that produced it are stored alongside it. Huyen also argues that aggregate accuracy alone is insufficient for production readiness; she advocates perturbation testing (checking whether small input changes cause disproportionate score shifts) and slice-based evaluation (decomposing the aggregate score by data subgroup) as the tests that reveal whether a stored run's headline number is trustworthy.

Practice then folded eval runs into the deployment📤DeploymentEngineeringA deployment eventView reference → pipeline. The widely repeated rule in modern AI engineering is that prompts are code, which makes a regression eval the natural gate: run the suite on every prompt or model change, block the change if the score drops. The run stops being a one-off experiment and becomes a recurring check, like a unit-test job🔨JobUserJob To Be Done: what the user is trying to accomplishView reference →.

Huyen makes the production-gate logic explicit: she argues that the validation split — not the held-out test set — should bear the weight of iterative tuning, precisely so that the test run's score remains an uncontaminated reading against a fixed instrument. That separation maps directly onto the eval_run model: runs against the benchmark accumulate on the validation distribution during development; the benchmark itself stays fixed, and a final run against it functions as the release🚢ReleaseProduct SpecificationA shipped version of the productView reference → gate.

How it works in practice

A team ships a summarisation feature⭐FeatureProduct SpecificationA product capability or featureView reference → with a 120-example offline eval, each example paired with a reference summary and a judge rubric scoring faithfulness and concision. The baseline eval run on prompt version 4 scores 8.1 out of 10. An engineer rewrites the system prompt to cut verbosity and opens a pull request. The CI regression eval fires automatically: prompt version 5 scores 8.4 on concision but 7.2 on faithfulness, because the tighter prompt started dropping caveats. Two stored runs, side by side, make the trade-off legible, and the change is held until the faithfulness drop is fixed. Without the recorded baseline run, the regression would have reached users as a quiet quality decline.

Eval run vs. its neighbours

• Eval benchmark🎖️Eval BenchmarkAI & Machine LearningA benchmark for evaluating AI qualityView reference → is the reusable instrument: the dataset plus the scoring rule, held constant. The run is one execution of that instrument. You author a benchmark once and accumulate many runs against it.
• Metric is the number a run produces. A run is the event of measuring; the metric is the measurement. One run can emit several metrics at once, which is why they are separate entities.
• AI experiment👾AI ExperimentAI & Machine LearningAn AI-focused experimentView reference → is a structured investigation🔍InvestigationEngineeringAn investigation into an issue or incidentView reference → that may compare several runs to test a hypothesis🧪HypothesisValidationA testable belief about a solutionView reference →. A run is the atomic execution an experiment is built from; the experiment is the question, the runs are the readings.

In the graph

In the Unified Product Graph, eval_run🎟️Eval RunAI & Machine LearningAn evaluation run against a benchmark lives in the ai domain alongside ai_model🤖AI ModelAI & Machine LearningAn AI or ML model used within the productView reference → and eval_benchmark🎖️Eval BenchmarkAI & Machine LearningA benchmark for evaluating AI qualityView reference →. Its defining edge is eval_benchmark_executed_as_eval_runEval Benchmarkexecuted asEval Runhierarchy, which ties every run back to the fixed benchmark that produced it, and eval_run_produces_metricEval RunproducesMetriccausal, which records the scores that run emitted. Modelling the run as its own entity, distinct from the benchmark, is what lets the graph hold a benchmark's full history: every execution, against every model build, over time. That history is the raw material for the Model Evaluation Loop, where models are compared across runs rather than on a single snapshot.