Design Leakage-Safe Experiments

Create an experiment whose result would remain credible when the model meets new data in its intended operating environment. Leakage prevention starts with time and causality, not with a random split parameter.

Define the prediction event

Write down:

• entity and prediction grain;
• decision made from the prediction;
• prediction timestamp and feature cutoff;
• outcome definition and maturity window;
• eligible population at prediction time;
• deployment cadence, horizon, and retraining policy;
• operational cost of each error type.

Build an availability ledger for candidate features: source event, observation time, recording delay, correction or backfill behavior, earliest production availability, and whether the value depends on the outcome. Exclude or reconstruct features that would not exist in the same form at prediction time.

Read references/leakage-threats-and-splits.md when choosing a split or investigating a suspected leakage path.