run-iteration-eval
Installation
SKILL.md
run-iteration-eval
Measures a skill change by running the content cases in evals/content/v2/evals-v2.json through
claude -p and grading the outputs. Outputs land in skills-workspace/iteration-<TAG>/.
The runner and grader are split on purpose: running calls Claude and costs tokens; grading is pure regex Python and is free to re-run on outputs that already exist. Never re-run the runner just to re-score — re-grade instead.
Steps
- Sync the cache first — non-negotiable. The runner loads the skill from the plugin cache,
not
skills/. Run thesync-skill-cacheskill (or its script directly). If you skip this, the eval grades the previously-published skill and the entire run is wasted:bash .claude/skills/sync-skill-cache/scripts/sync-cache.sh