Using Counterfactual Statistics

Overview

When you fork an experiment into matched branches and compare them, you have bought precision, not independence. The branches are repeated measures of one thing; the thing is the run you forked. Nearly every statistical failure in this domain is a consequence of forgetting that — and the rest are consequences of choosing what to measure after seeing the results.

Counterfactual evaluation is unusually powerful and unusually easy to get wrong. Matching branches against a no-op control can cut required sample size by 6× (common-random-numbers-and-matching); counting those branches as independent samples understates your uncertainty by 2.6× and turns a null into a headline (statistical-units-and-clustering). The two errors live one line of code apart.

This pack covers the statistical discipline that paired counterfactual experiments demand and that general ML evaluation guidance does not: what the unit is, what makes the pairing valid, which data may inform which decision, how much of a "best-of-K" result is selection, and what a result must contain before anyone acts on it.

When to Use

Use this pack when: