Model Evaluation Report

A model that beats a naive baseline on aggregate accuracy can still be worse than useless in production. Honest evaluation means choosing the right metric before looking at results, comparing against baselines that actually matter, and finding the slices where the model fails - because those slices are where the incident reports will come from.

Operating procedure

Order matters: the primary metric and baselines are chosen before results are examined, or the report is p-hacked by construction.

Step 1: Gather inputs

Collect from the requester before computing anything; label unknowns as guesses.

1. Task type: classification, regression, or ranking.
2. The business decision the model drives, and the asymmetry of errors (is a false positive or a false negative more expensive, and roughly by how much).
3. The evaluation dataset: source, size, and confirmation it is disjoint from training data and from any data used for model selection.
4. What the model replaces: prior production model, rule-based system, or nothing.
5. Class balance (classification) or target distribution (regression).

Step 2: Fix the primary metric before looking at results