evaluation-design
Evaluation Design
Produce a measurement contract: a proposal stating which decision an eval informs, which behaviors and failures it reveals, on which cases, scored by which checks, under which conditions, and at what resolution. When building is requested, also produce the instrument in the project's runner. A large case set, a working runner, or a judge that agrees with itself is not by itself a sound measurement, and the designer does not certify its own work: a launch, hold, or trust verdict belongs to evaluation-audit, which reads the artifacts rather than this skill's summary.
A design request returns a proposal. Writing cases, labels, judges, or runner code needs the request. Fast checks within the always-on tier may proceed. Present spend beyond the always-on tier or the project's stated budget first, with one sentence predicting the result; a praxis profile's compute ladder defines the tiers when present. Use the project's runner, records, and schemas; do not create a new harness or tracking framework. If the request turns out to be a runner fix, a status question, or a verdict on an existing eval, say so and hand it to its owner.
The decision
Name who acts on the result and what they decide: launch a campaign, adopt a model or configuration, change a prompt, report to a customer, or support a research claim. Then name the smallest difference that would change that decision, in the eval's own unit, such as the margin a candidate must beat the incumbent by before a switch. The rest of the design is sized against it. If nobody can name the decision or the difference, record the eval as exploratory: its numbers can describe behavior but cannot gate a choice.
Name the unit being scored (case, trial, episode, conversation, task completion) and keep counts in it. Report task success, policy compliance, quality, cost, and latency apart when one score would hide a trade-off.
Real outputs before failure categories
Read real outputs of the system under test before fixing failure categories. A deployed system has production traces; a new capability has pilot trajectories, and a few pilot runs on draft cases are enough to start. Note what went wrong in each output, then group the notes into failure modes. Categories from a taxonomy, a prior benchmark, or a generic quality list are hypotheses until outputs show them, so record which modes were observed and which are only proposed. Sample at random as well as by outliers or clusters. A failure that exists because the prompt never asked for the behavior is a specification fix before it is an evaluator.