Statistical Analysis Planner

Design the statistical analysis before running, and report it correctly after results exist. This skill prevents underpowered claims, misleading averages-without-variance, and significance theater in ML papers.

Use this skill when:

• deciding which significance tests or confidence intervals to report for a result table
• seed variance is high and a single-run result may not be representative
• comparing methods and wanting to know if the difference is statistically meaningful
• a paper or rebuttal needs to defend a claim quantitatively against reviewer variance concerns
• an ablation result is close and the decision to include it depends on whether the difference is real
• multiple comparisons are being made and type-I error accumulation needs to be controlled

Do not use this skill to run the experiments — use run-experiment. Do not use this skill to interpret surprising results scientifically — use result-diagnosis. Use this skill after results exist (or in planning mode before deciding how many seeds to run).

Pair this skill with: