Validate Judge

Measure whether a judge prompt agrees with human labels.

Lifecycle

Setup → Design → Calibrate → ► Validate → Execute → Diagnose → Iterate

This skill follows generate-judges. It produces confidence that the judge works before you run kensa eval at scale. Skip this if your judge uses only deterministic checks.

Why TPR and TNR, not accuracy

Accuracy is misleading when classes are imbalanced. If 90% of outputs pass, a judge that always says "pass" gets 90% accuracy but catches zero failures.

• TPR (true positive rate): of outputs humans labeled pass, what fraction does the judge pass?
• TNR (true negative rate): of outputs humans labeled fail, what fraction does the judge fail?