Evaluate Skill

Run a skill repeatedly to measure how reliably it works, and design the evals that measure it.

Prerequisites

The caliper CLI must be on PATH. This skill can be copied into an agent without the Caliper repo, so do not assume the CLI is packaged with it. Install if missing:

pipx install caliper-eval

The engine (backend + model) is not part of the spec — it is chosen at run time with --model (skill) and --judge-model (judge), independently, from claude-code, codex, pi, defaulting to claude-code. Every backend is a CLI agent that uses its own subscription/auth; there is no direct-API backend (for API billing, configure a CLI with an API key). Full per-backend detail and every command: REFERENCE.md.

Spec shape

An .eval.yaml names the skill and a list of tasks. Keep skill.path relative to the spec file (usually ./SKILL.md):