iterate-evaluator
Iterate on an evaluator
Evaluators are versioned. Tuning a judge doesn't overwrite it — you add a new
version and (usually) make it the live one. Old versions stay for
comparison and rollback. Drives the calibrate evaluators commands. Ask at each
step; if $ARGUMENTS already carries a UUID or name, pre-fill it and skip the
lookup.
Built-in default evaluators are editable too, not just ones you authored — tune them with the same flow below. So "the judge is wrong on X" is fixable even when the judge is one of the defaults.
Some evaluators are locked (is_protected in evaluators get): they can't
be deleted, and only their name, description and rubric can change. If the
user wants something else changed on one, say it's locked and offer to make a
copy as a new evaluator instead.