Evaluation Reality Check

Mission

Design evaluation strategies that match task type, deployment setting, time, groups, and decision risk.

Primary failure mode prevented: offline evaluation often overstates real-world performance because splits, metrics, and benchmarks do not match deployment.

When To Use

• The team needs a structured go | fix-first | stop | proceed-with-risk | needs-owner-review decision before committing engineering or modeling effort.
• Local artifacts (task type, data generation process, deployment use, target and features) must be audited for red flags such as random split for temporal deployment, test set reused for tuning, no external validation for domain shift, single aggregate metric.
• Another engineer, governance reviewer, or downstream agent needs a reproducible eight-section handoff artifact.

Operating Rules