Eval Dataset Design

Your evals are only as good as the dataset they run on. Miss a user scenario and you'll never catch regressions on it.

When to Use

• Starting an eval program from zero
• Your evals pass but users still hit issues → coverage gap
• Labels are inconsistent across reviewers → quality problem
• Adding evals for a new feature or domain

Dataset Properties Worth Optimizing

1. Coverage — representative of real user queries
2. Difficulty distribution — mix of easy/medium/hard, not all easy
3. Label consistency — two humans agree on the label
4. Stability — same inputs → same evaluable outputs over time
5. Uncontaminated — not in the model's training data