Validate Evaluator

Calibrate an LLM judge against human judgment.

Respond in the user's language; keep metric names (TPR/TNR) and code in English.

Overview

1. (Optional) Label traces with the user if labels don't exist yet
2. Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
3. Run judge on dev set and measure TPR/TNR
4. Iterate on the judge until TPR and TNR > 90% on dev set
5. Run once on held-out test set for final TPR/TNR
6. Apply bias correction formula to production data

Prerequisites