llm-evals
LLM Evals
Prove a criterion about LLM output with an eval: an approved case set, a runner that calls the product's entry point, and a grader, scored over repeated runs. It is a test in a noisy medium. The bar exists before the behavior changes, the baseline score is the red, and the green is a held-out score at or above the target and above the baseline outside the noise floor.
An eval is one of two kinds. A capability eval starts below the target so there is a hill to climb, and it is finished once it reaches green. A regression suite holds the cases a capability eval finished with, at the level they reached, so a score below that level means something broke rather than that there is room to improve. Every criterion starts as a capability eval and ends by graduating into the project's one regression suite.
Read GLOSSARY.md, then use docs/decisions/README.md to load only the
decision subjects relevant to the behavior under evaluation. Surface a conflict
between code, spec, and a decision contract instead of choosing silently.