Calibrate an evaluator against humans

An LLM judge is only trustworthy once it agrees with people. This is Calibrate's signature loop: label a sample by hand, have the judge label the same sample, measure agreement, and tune the judge until the gap closes. Drives the calibrate annotation-tasks and evaluators commands.

References: ../../references/agent-mode.md (output/polling), ../../references/config-shapes.md (item + evaluator shapes). Keep what you say to the user plain — see ../../references/voice.md.

Get the latest instructions

These instructions are updated often, and the copy installed on this machine can be months old. Before doing anything else, pull the latest copy of this skill: