Data Drift Monitor

Models trained on historical data degrade when the world changes. Drift monitoring is the early-warning system that distinguishes a model aging gracefully from one silently failing - and without thresholds tied to actions, it degenerates into dashboards nobody reads and alerts everybody mutes. This skill produces a monitoring plan where every alert has a named owner and a defined next step.

Operating procedure

Work in order: drift types (Step 2) determine which tests apply (Step 3), and thresholds (Step 5) are meaningless until cadence and coverage (Step 4) define what is measured.

Step 1: Gather inputs

1. The model's top features by importance (at minimum the top 10) and their types (continuous vs categorical).
2. Prediction volume and frequency - this sets monitoring cadence.
3. Ground-truth label lag: how long after a prediction the true label arrives. If unknown, label the estimate a guess.
4. The primary business metric the model serves and its current 30-day baseline.
5. Retraining cost in engineering time and compute, so triggers can be calibrated against it.

Step 2: Classify what can drift

Not all drift requires the same response.