LLM-as-Judge

Use a strong LLM to evaluate another LLM's output. Done right, it's fast, cheap, and correlates with human judgment. Done wrong, it's biased, inconsistent, and misleading.

When to Use

• Scaling eval beyond what humans can review
• Measuring open-ended outputs (summaries, code quality, helpfulness) where rule-based metrics fail
• Pairwise model comparison (A vs B on the same input)
• CI checks on agent outputs

When NOT to Use

• High-stakes decisions (medical, legal) — need humans
• When the judge is the same model as the generator — biased toward its own style
• Very short outputs where a rule can decide — exact_match is cheaper
• Tasks the judge can't do itself — if it can't write good code, it can't judge code well

Three Common Patterns