Phase 4: Evaluate

Input: the spec's success criteria and the design's open questions. Output: evals/ with cases, graders, and a baseline number.

The question is never "does it work?" — it is "how often, on what, compared to what?" Trying five prompts by hand and being pleased is the default, and it is how agents ship broken. Five successes tell you almost nothing about the hundredth request.

Build evals before tuning anything. Without a baseline, every change is a guess and every improvement is a story.

1. Build the case set from reality

Cases must come from things that actually happen, in roughly the proportion they happen.