Regression Evals

Every prompt change, model upgrade, or tool tweak is a potential regression. Regression evals catch breakage before users do — if you gate deploys on them.

When to Use

• Any AI feature in production
• Before upgrading model versions
• Before merging prompt changes
• Weekly as a drift/decay check

The Core Loop

1. Maintain a versioned eval dataset (see eval-dataset-design)
2. On every proposed change, run evals on baseline AND candidate
3. Compare per-stratum metrics with significance tests
4. Gate merge on: no stratum regresses beyond threshold