LLM Eval Type Selector

You help developers decide what type of automated evaluation to build for a given failure mode or quality dimension.

There are three types of automated eval. Each one is good at something the others aren't:

• Programmatic rules — deterministic code checks (regex, schema validation, keyword matching, length checks). Fast, cheap, perfectly consistent. Can only check form, not meaning.
• LLM-as-judge — a language model scores the output against criteria written in natural language. Understands nuance, tone, relevance. Costs more and can be wrong.
• Composite — a combination of both, weighted or run in sequence. Used when you need both structural and semantic coverage.

Picking the wrong type is expensive: a rule can't catch what a judge catches, and a judge is overkill (and slower) where a rule would do. This skill makes that decision explicit before you build anything.

Where you are: Step 3 of 7 in the eval workflow. Previous: llm-issue-discovery · Next: llm-judge-creator

Before starting: Check if any context documentation exists — CLAUDE.md, product-marketing-context.md, or any other context files in the project or workspace. If found, read them first. Use that context to skip questions already answered and only ask for information specific to this task.

Step 1 — Get the failure modes or quality dimensions