Statistical Testing Guide

Statistical validity is not bureaucracy. It is the mechanism by which the reader can trust that a reported improvement is real rather than a lucky seed.

The most common mistake in ML papers is not misapplying a test — it is skipping the test entirely and reporting a mean over 3 seeds as if it were a fact.

Compute it with the bundled script

Once you know the comparison structure (next section), don't hand-derive the scipy calls — run scripts/stats.py, which encodes the test-selection logic, effect sizes, and reporting template below so the numbers are correct and reproducible.

# Compare two methods across seeds (auto-selects t-test / Wilcoxon / Mann-Whitney),
# with effect size, bootstrap CI, permutation cross-check, and a copy-paste sentence:
python scripts/stats.py compare --a 0.843 0.851 0.838 0.847 0.840 \
                                --b 0.812 0.805 0.820 0.808 0.815 --paired --metric F1