llm-eval-anti-patterns

An LLM eval suite can be green and still be worthless. The failure is not in the system under test, it is in the eval definition: too few cases to distinguish signal from sampling noise, assertions that cannot fail for the right reason, no baseline to compare against, or a judge model whose verdicts have never been checked against a human.

This catalog reviews the eval definition, not the eval run. Read the config, the test data, the assertions, and the CI job that executes them. Every finding must point at a line in one of those.

What this owns, and what it does not