Advanced Evaluation

Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.

When to Activate

• Building LLM-as-judge systems for LLM outputs
• Comparing multiple model responses to select the best one
• Establishing consistent quality standards across evaluation teams
• Debugging evaluation systems that show inconsistent results
• Designing A/B tests for prompt or model changes
• Creating rubrics specifically for LLM or human/LLM hybrid judges
• Analyzing correlation between automated and human judgments

Do not activate this skill for adjacent work owned by other skills:

• General deterministic checks, regression suites, production quality gates, or outcome metrics: evaluation.
• Tool API contracts for evaluation tools: tool-design.

Core Concepts