agent-observability-build-eval-from-annotations
build-eval-from-annotations — fit an evaluator to human labels, by measurement
Humans label the first rows of an annotation queue; this skill turns those labels into an evaluator
that can label the rest. It is the measured version of that idea: the judge is never "written
and shipped", it is fitted — scored against the human labels, its errors read, changed once,
re-scored, kept only if it improved. Same control loop as
agent-observability-auto-experiment, different object under optimization: there the hill-climb
edits the app, here it edits the judge, and the ground truth is a human's label rather than a
rubric.
Read references/rubrics.md in full before iteration 1 and keep it in mind every iteration. It
holds the non-negotiable rules (never invent a label; what may count as evidence; the metric floor;
the degenerate-judge guard; the publish gate). This file is the control loop; that file is the law.
Related skills: agent-observability-eval-bootstrap proposes evaluators from unlabelled traces by
inspection — use it when there are no human labels. This skill is for when there are labels, and
they are the thing being fitted to.