AI Experiment

Origin & evolution

The discipline is offline evaluation, imported from machine learning🧠LearningValidationAn insight gained from an experimentView reference → and adapted to the open-ended outputs of language models. Where a classifier could be scored against labelled ground truth, generated prose resists exact-match grading. The field's answer was LLM-as-a-judge: use a strong model to score another model's output against a reference or a rubric. Evidently AI and LangSmith both describe the now-standard loop, where an experiment runs▶️Experiment RunValidationAn execution instance of an experiment that records actual conditions, observations, and raw results.View reference → a candidate system over a curated dataset and a judge or metric produces comparable scores. Mature practice keeps a held-out slice of manually labelled examples to validate the judge itself, since an unchecked judge inherits its own biases.

How it works in practice

A team wants to swap a prompt that is costing too many tokens. They assemble a dataset of 200 real support questions with hand-written ideal answers. They run two configurations: the live prompt and a leaner candidate. An LLM judge scores each answer for accuracy and helpfulness; a script measures token use. The candidate scores 4.4 against the live prompt's 4.5 on quality, a difference inside the noise, while cutting tokens by 38%. The experiment🧫ExperimentValidationA test designed to validate a hypothesisView reference → makes the trade-off legible, and the team ships the cheaper prompt with confidence rather than crossing their fingers in production.

AI Experiment vs. its neighbours

• AI Trace🪡AI TraceAI & Machine LearningAn AI inference traceView reference → records what one production call actually did. An experiment runs many controlled calls against a known dataset before anything reaches users. The trace is observation👁️ObservationUser ResearchA specific behaviour or statement observedView reference → in the wild; the experiment is a lab test.
• Eval Benchmark🎖️Eval BenchmarkAI & Machine LearningA benchmark for evaluating AI qualityView reference → is the standardised yardstick (a public test set and metric) that an experiment can run against. The benchmark is the ruler; the experiment is the act of measuring your system with it.
• AI Dataset🪆AI DatasetAI & Machine LearningA dataset for AI training or evaluationView reference → is the fixed input an experiment runs over. The dataset is the question set; the experiment is the graded sitting.

In the graph

In the Unified Product Graph, an AI Experiment lives in the AI region as the evaluation primitive. The model under test connects through ai_model_evaluated_through_ai_experimentAI Modelevaluated throughAI Experimenthierarchy, and any models the experiment invokes (a judge, a baseline) connect through ai_experiment_uses_ai_modelAI ExperimentusesAI Modelcross-domain. Keeping the experiment as its own node means a model's quality claims point at the evidence that backs them, so "we improved accuracy" is always traceable to the run that demonstrated it.