Credo: Evaluation

Establish whether an evaluation measures the intended capability and supports the decision being made. A working runner and a high score do not establish benchmark validity.

Define the measurement

Recover the user's question, actual system use, target population, inputs and outputs, available data, and existing evaluator. State what ability or failure the measurement should reveal and which decision it informs. Do not replace the question with the easiest available metric.

Identify the unit being evaluated: a response, case, conversation, task completion, or system interaction. Separate task success, policy compliance, quality, latency, cost, and other dimensions when combining them would hide a consequential tradeoff. Use an aggregate only when its meaning and weighting fit the decision.

Design or inspect the method

Read measurement and scoring for metrics, judge validation, uncertainty, and benchmark implementation. Select the checks needed for this question:

• Relate cases to intended behaviors and coverage. Include boundaries and plausible failure modes, not just easy examples or a leaderboard convention.
• Define scoring from observable outputs. Preserve units, denominators, exclusions, missing outcomes, and failure treatment in the underlying method.
• Choose comparison conditions that make the result meaningful, including model version, prompt, tools, decoding, available information, and resource budget when they affect the claim.
• Inspect shortcuts, contamination, and ways to improve the score without improving the intended capability. A suspected shortcut calls for a test, not a claim that the benchmark is invalid.
• Separate exploratory choices from assessment on material kept independent of those choices.