Eval Best Practices

Design or audit the measurement program an AI agent or skill is judged by. Run to completion, stating assumptions; ask only on a genuine blocker such as no obtainable ground truth or no way to observe the intended outcome.

Core principle: A trusted-but-wrong instrument is worse than none. Start from the product decision and externally verifiable user outcome, then decompose failure modes and spend judge budget only on the subjective residual.

Invariants

• Task Fidelity: an impossible, ambiguous, leaky, or state-contaminated task measures the harness, not the agent. Validate the task, verifier, information boundary, and clean environment before diagnosing the system.
• Outcome Truth: verify the achieved external state first. Grade the path only where authorization, safety, required hand-offs, cost, latency, or retry behavior can hide behind success; never require one exact path unless that path is itself an invariant.
• Vague Judge: define the positive class and measure the confusion matrix against human-labeled failures and passes. Calibration belongs to (prompt × model × rubric); changing any one invalidates it.
• Stochastic Evidence / march of nines: per-step or single-run success is not end-to-end reliability. Repeat isolated run-level trials using the deployed retry model; report counts, uncertainty, slices, and baseline deltas. Treat p^N as a planning warning, not observed reliability.
• God Gate: one aggregate hides which risk broke. Classify each metric as blocker, thresholded guardrail, north star, or diagnostic, with an owner and failure path.

Workflow