experiment-design
Overview
Based on "The Art of Statistics" by David Spiegelhalter. The core principle: statistical significance is not the same as practical significance, and a poorly designed experiment produces confidently wrong answers. Rigorous experiment design means defining the question, estimating the required sample size, and setting success criteria before any data is collected - then interpreting results within the limits of what the experiment can actually prove.
Workflow
Step 1: State the hypothesis in falsifiable form
A hypothesis must specify: the intervention, the population, the outcome metric, and the expected direction.
Format: "Applying [intervention] to [population] will [increase/decrease] [metric] compared to the control condition."
Example: "Showing personalized recommendations to new users in the first session will increase 7-day retention compared to showing trending content."
Also state the null hypothesis explicitly: "There is no difference in 7-day retention between personalized and trending content groups."
Spiegelhalter's warning: if you cannot state what result would cause you to reject your hypothesis, you do not have a hypothesis - you have a belief.