Counterfactual Statistics

Use this contract for the concrete task. Apply a short relevant check for a small change; expand investigation when the failure, risk or requested artifact warrants it. Resolve facts from the repository and runtime before imposing a process. Delegation is optional and should answer a bounded unresolved question.

Experiment contract

• Name the independent generating unit and the estimand before counting observations. Branches, checkpoints and repeated decisions from one run are dependent measurements.
• Record what matching preserves: starting state, future data, augmentation and RNG streams. Matching improves precision; it does not create independent runs.
• Separate data used to construct candidates, select winners, calibrate admission and report results. Split whole generating units; record any cross-fitting scheme.
• Define the no-intervention comparator and cost accounting. A zero-anchored utility is useful when doing nothing is an available decision; do not impose it on unrelated estimands.
• Choose endpoints, horizons, exclusions, testing family and stopping rules before confirmatory data is inspected. Label subsequent changes exploratory.
• Size the design using unit-level variance and a practically meaningful effect. Report uncertainty in pilot variance, not a fleet size inferred from a selected winner.

Evidence and output

Produce a compact analysis plan or result containing the unit count, pairing/split contract, estimand, selection history, analysis assumptions, interval and limitations. Include practical effect, cost and failures/abstentions where relevant. Store null and negative results. Aggregate or model clustering explicitly; a paired test alone does not correct correlated rows.

A clean audit is valid when the relevant checks are supported by evidence. Distinguish a confirmed defect, a robustness concern and a check that could not be assessed. Never infer that a pipeline is defective because no finding was produced.