Create Judge

Guide users through designing and creating an automated judge that evaluates LLM outputs in ZeroEval.

When To Use

• Creating a new judge from scratch for any evaluation goal.
• Deciding between binary (pass/fail) and scored (numeric rubric) evaluation.
• Writing judge templates (the evaluation prompt the judge model runs).
• Designing structured criteria for multi-dimensional scored judges.
• Linking a judge to a specific prompt for automatic feedback.
• Troubleshooting judges that aren't producing expected evaluations.

Execution Sequence

Follow these steps in order. Load reference files only when needed for the current step.

Step 1: Understand the Evaluation Goal