Eval-Driven Development

Concept of the skill

Eval-driven development is the practice of building LLM-integrated systems by writing evaluations before and alongside the system, with each eval defining a behavioral criterion the system must satisfy on a representative input set, and the suite's aggregated signal gating every change to prompt, model, retrieval, context assembly, scaffolding, tooling, or orchestration. Five primitives: (1) eval dataset — curated input examples representing production plus edge cases, prior failures, and held-out slices, typically JSONL of {input, reference} records checked into version control; (2) evaluation function — per-example grader producing a score, with mechanism chosen from programmatic, model-graded, human-graded, preference comparison, trace check, or hybrid; (3) aggregation — statistical summary across the dataset (pass-rate, weighted pass-rate, stratified slice scores, distribution) with sample size and either a confidence interval or a defined minimum-detectable-change threshold, plus cost/latency side metrics; (4) iteration loop — eval → diagnose → change one meaningful variable → re-eval → compare, with the suite stable while the system changes; (5) regression budget — per-eval/per-slice merge policy: gating (no regression allowed), optimizing (improvement gates merge), watchful (tracked, not gated), human-signoff, or cost-aware threshold (a pass-rate gain only counts if it clears the eval's cost/latency budget — a 1pp quality gain at 3× inference cost is a regression).

Replaces "ship LLM systems based on how the model feels at the desk" (and benchmark shopping) with empirical measurement that distinguishes "the new prompt works better" from "the new prompt works better on the five examples I happened to try." Solves the problem that LLM system behavior is stochastic (every run is a sample, not a deterministic pass/fail), the input space is open-ended (the developer's pet examples are not a representative sample of what users will throw at it), model behavior changes across releases (a feature that failed last quarter may now be viable), and traditional binary unit tests do not capture the distributional nature of LLM outputs. The eval suite is the empirical measurement instrument that makes shipping decisions defensible and changes gateable: each change has a baseline, a representative private eval set, a comparison method, and a documented decision policy. The discipline's hard part is not writing evals — it is choosing what to measure, encoding the choice into a grader the team agrees with, sampling a dataset that represents production, distinguishing a real improvement from sampling noise, tracking high-risk slices separately from the average, calibrating model-graders against humans, feeding sanitized production failures back into the next eval version, and resisting Goodhart's Law as the eval suite becomes the optimization target. Teams that get this right ship systems whose quality matches their stated definition of "good"; teams that get this wrong ship systems that ace evals and disappoint users.

Distinct from testing-strategy, which owns deterministic-software testing where every run is binary pass/fail and the main decision is the cheapest honest test level — this skill owns LLM evaluation where every run is a sample from a distribution and pass-rate (or a score distribution) is the unit of judgment; the disciplines share vocabulary (suite, gate, regression) but the math underneath differs. The exact construction of eval task sets, rubrics, graders, hard negatives, trace schemas, and thresholds is separate from this skill; this skill owns the development discipline that uses those artifacts to gate whether prompt/model/retrieval/tool/orchestration changes ship. Distinct from evaluation, which owns general scoring frameworks and result interpretation — this skill owns the repeated LLM-system change gate. Distinct from tool-call-flow, which owns the protocol cycle by which a model invokes tools — this skill owns the discipline of measuring whether that cycle produces correct behavior; tool-call and trajectory evals are a specialization of the general pattern. Distinct from prompt-injection-defense, which owns the security property and threat model — this skill owns the measurement of whether the property holds (red-team evals against an injection corpus are one application). Distinct from error-tracking and observability-modeling, which own runtime measurement of deployed systems — this skill owns offline pre-deployment measurement while using production signals as inputs for dataset refresh; one is not used as a substitute for the other. Eval-driven development is to LLM system engineering what crash-test ratings are to automotive safety — you do not ship a car based on how well it parked in your driveway; you ship it after a battery of standardized tests on representative crash scenarios, with the pass-rate against named criteria as the gating signal. A score of 4.3 stars across the suite is the only defensible claim of 'safer'; a developer's intuition that 'the new model feels smarter' is the unmeasured equivalent of 'I drove it home, it seemed fine.' And just as a crash lab that the manufacturer trains its cars to detect stops measuring real safety, an eval the model can recognize as a test stops measuring real behavior. The wrong mental model is that LLM evals are "unit tests for prompts" — binary pass/fail, written once, checked in CI, ship on green. They are not. The fundamental property is distributional and lifecycle-based: every run is a sample from a stochastic system, and the unit of judgment is pass-rate over a sampled population with a sample size and confidence interval. A test that runs once and "passes" tells you nothing about the distribution; pass-rate at n=20 has huge uncertainty; pass-rate at n=500 starts to be informative. Adjacent misconceptions: that a higher headline pass-rate is a real improvement (it is not until the delta's confidence interval excludes zero — a paired bootstrap on a shared dataset is the cheap default, McNemar's test is the right check for paired pass/fail outcomes, and halving the effect you want to detect costs four times the samples); that a single headline number is sufficient (it is not — it can hide a high-risk slice regressing while the average improves; a panel of independent measures and slices is harder to over-fit, and HELM's multi-metric framing exists precisely as a counter to single-metric Goodharting); that public benchmarks (MMLU, HumanEval, BIG-bench, GAIA, MT-Bench, τ-bench) gate system-specific shipping decisions (they do not — benchmarks predict how a model will do on the exact tasks they contain, not on your system's user inputs; use them for model-selection grounding, and treat a sudden benchmark jump as a contamination hypothesis, not an automatic capability win); that higher eval scores are always improvements (they are not — Goodhart's Law: when the eval becomes the optimization target, it ceases to be a good measure; a 100% pass-rate is a measurement failure on a capability/gating suite, not a victory, because a suite the system aces has stopped discriminating — though a regression suite near 100% is doing its job, see the carve-out in § The 100% Pass-Rate Trap); that model-graded evals are reliable without calibration (they are not — model-graders have verbosity bias, position bias in pairwise comparisons, self-preference bias toward their own family's outputs, and correlated error with the system being graded; the fix is mechanical — judge from a different top-tier family, swap positions, ensemble across families — not a "be objective" instruction in the grader prompt, because the bias lives in the embedding distribution, not the surface style); that you can trust a metric without reading the transcripts (you cannot — error analysis on the actual traces is how you learn whether the grader itself is working and what the failures really are); that final-output grading is sufficient for multi-step agents (it is not — an agent that reaches the right answer through a broken or unsafe trajectory will pass a final-output eval and fail in production; multi-step systems need trajectory-aware evals that score planning, tool selection, and execution); that the model under test is fixed (it is not — every model upgrade or vendor switch changes the system under test and must be re-gated against the private suite, including high-risk slices, cost, and latency); and that offline evals replace production telemetry (they do not — production telemetry measures the actual user-facing system under actual load, and every production incident is a candidate eval case after sanitization; the two are complementary, not substitutable). Public benchmarks, vendor dashboards, and LLM judges are useful inputs, not shipping authority by themselves.

Coverage

The practice of building language-model-integrated systems by writing evaluations before and alongside the system, then using the eval suite's measured signal to gate every meaningful change. Covers: