iterate-evaluator

Installation
SKILL.md

Iterate on an evaluator

Evaluators are versioned. Tuning a judge doesn't overwrite it — you add a new version and (usually) make it the live one. Old versions stay for comparison and rollback. Drives the calibrate evaluators commands. Ask at each step; if $ARGUMENTS already carries a UUID or name, pre-fill it and skip the lookup.

Built-in default evaluators are editable too, not just ones you authored — tune them with the same flow below. So "the judge is wrong on X" is fixable even when the judge is one of the defaults.

Some evaluators are locked (is_protected in evaluators get): they can't be deleted, and only their name, description and rubric can change. If the user wants something else changed on one, say it's locked and offer to make a copy as a new evaluator instead.

Installs
21
First Seen
Jul 13, 2026