Diagnosing ML Failures

Turn a vague failure into a minimal, testable discrepancy and isolate the first layer where expected and observed behavior diverge. Do not begin with hyperparameter tuning: it can hide the fault while destroying causal evidence.

Define the symptom

Capture one concrete failing example or comparison:

• expected versus observed behavior;
• first known bad and last known good run, version, or time window;
• affected population and unaffected control population;
• metric implementation, threshold, and aggregation level;
• exact input, artifact, code revision, configuration, and environment when available.

Reproduce the symptom through the smallest stable entry point. If it is intermittent, estimate frequency and identify what varies between runs before changing the system.

Isolate the first broken layer

Trace the same records through boundaries in causal order: