LLM-as-Judge

A judge is a model, and models are biased. Calibrate it like any other classifier.

LLM-as-judge is the only scalable way to score open-ended outputs — and the easiest way to ship a "passing" agent that's actually worse. Position bias, length bias, self-preference, and verbosity reward are routine. Treat the judge as a system that needs its own eval before it's allowed to score anything.

When to use this skill

• The user is scoring open-ended agent output (summaries, drafts, plans, refactors).
• The user is using one LLM to compare two LLMs.
• The user is reporting eval results without confidence intervals or judge-agreement numbers.
• The user is using the same model as both candidate and judge.

The four bias modes

Every LLM-judge has at least the first three. Name them; correct for them.