Imbalanced Data

Overview

When positives are rare, naive training and naive metrics both mislead. A model that always predicts "negative" can score 99% accuracy and catch zero fraud. Handle imbalance at three levels: metric, algorithm, and threshold — and resample inside cross-validation, never before.

When to use

• Class ratio is skewed (e.g., 95/5 or worse).
• Catching the rare class matters (fraud, defaults, rare disease, defects).

Step 1 — Fix the metric first

Drop accuracy and ROC-AUC-as-sole-metric. Prefer:

• PR-AUC (average precision) — most informative for rare positives.
• Recall @ fixed precision — "catch X% of fraud while keeping false alarms tolerable."
• F1 / Fβ — β>1 weights recall when misses are costly.