ML Evaluation

Overview

Use this skill for evaluating machine learning systems before, during, and after deployment. A valid evaluation answers: what decision will use the model, what data distribution matters, what failure costs exist, which metric encodes those costs, and whether the measured improvement is real rather than leakage, variance, or overfitting to validation.

Evaluation Design Checklist

1. Define the unit of prediction and the unit of independence. Split by user, account, patient, device, document, session, or time when examples are correlated.
2. Establish baseline performance using a simple heuristic, prior model, or linear/tree baseline.
3. Pick metrics aligned to the decision, not just convenient defaults.
4. Freeze test data until final assessment; use validation data for tuning.
5. Evaluate slices: labels, geography, device, language, demographic groups when appropriate, data source, time period, and known hard cases.
6. Report uncertainty with confidence intervals or repeated CV when sample size is limited.
7. Perform qualitative error analysis before scaling model complexity.

Split Strategies

Use random stratified splits only for iid data where examples are independent. Use grouped splits when multiple rows share an entity. Use time-based splits for forecasting, recommendation, fraud, logs, and any deployment where future data differs from past data. Use nested cross-validation when both hyperparameters and performance estimates must be unbiased. For small datasets, repeated stratified CV can reduce variance, but keep a final untouched test set when stakes are high.