LLM-as-Judge

Goal

Score open-ended LLM and agent outputs — where no single reference answer exists — by using another LLM as the evaluator, while actively controlling for the judge model's well-documented biases. An LLM judge is fast, scalable, and cheap relative to human review, but it is biased-but-useful, never ground truth. The entire discipline is: extract signal from a flawed instrument by constraining it and calibrating it against humans.