Write LLM-as-Judge Prompt

Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.

Respond in the user's language. The judge prompt itself is normally written in English, or in the language of the traces it will evaluate.

Prerequisites

• Error analysis is complete. The failure mode is identified.
• The user has human-labeled traces for this failure mode (at least 20 Pass and 20 Fail examples).
• A code-based evaluator cannot check this failure mode. Exhaust code-based options before reaching for a judge — many failure modes that seem subjective reduce to keyword checks, regex, or API calls when you understand the domain. Example: detecting whether an AI interviewing coach suggests "general" questions (asking about typical behavior instead of a specific past event) seems to require semantic understanding, but in practice a keyword check for words like "usually," "typical," and "normally" could work quite well.

Cold start: no labeled traces yet

If labeled traces do not exist: STOP. Collect and label 10-20 real traces with the user first (its companion skill validate-evaluator, if installed, includes a minimal labeling flow for this); full calibration later needs ~100 labeled traces (see validate-evaluator). Never fabricate few-shot examples — fabricated examples violate the training-split rule and produce an ungrounded judge: it will agree with your invented notion of the failure mode, not with how the failure actually appears in production.

The Four Components

Every judge prompt requires exactly four components: