Agent Test

Design the measurement an AI agent or skill is judged by — eval suites, LLM judges, trajectory tests, held-out benchmarks, activation evals. Run to completion, stating assumptions; ask only on a genuine blocker (no traces, no obtainable ground truth).

Core principle

A trusted-but-wrong instrument is worse than none. Name the failure modes before the aggregate — you cannot improve a number you cannot decompose — and calibrate the judge before it gates. Prefer a deterministic check to a judge wherever the property is checkable (schema, exact match, tool-arg shape, exit code); spend judge budget on the subjective residual only.

Inlined invariants

• Vague Judge: a judge with no precision/recall against a human-labeled set is an opinion machine, not an instrument. Calibration is a property of (prompt × model × rubric) — changing any one invalidates it.
• Trajectory Blindness / march of nines: per-step grading misses path failures, and a 0.95 per-step bar compounds toward 0.95^N over an N-step run. Gate on the run-level rate.
• God Gate: one aggregate pass-rate hides which slice broke. Tag slices guardrail (block ship) vs north-star (report only).
• Goodhart: a metric that becomes the target stops measuring; a benchmark you tune against silently rejoins the training set.
• Skeleton fixtures — metadata stubs without a real prompt + completion + tool I/O — certify nothing; reject them before they seed a suite.

Workflow