Experimentation

Growth experimentation is the scientific method applied to software conversion funnels. A/B testing fails when teams launch underpowered tests, declare premature victory without statistical significance, or test trivial cosmetic changes. A professional experimentation practice requires statistical sample-size power calculations, structured hypothesis backlogs (ICE scoring), and disciplined holdout verification.

1. Statistical Foundations & Test Sizing

Before launching an experiment, calculate mandatory sample size based on statistical parameters:

• Minimum Detectable Effect (MDE): The smallest relative conversion lift you care about detecting (e.g. 10% relative lift).
• Statistical Power ($1 - \beta$): Standard baseline is 80% (20% false negative rate).
• Statistical Significance ($\alpha$): Standard baseline is 95% ($p < 0.05$, 5% false positive rate).
• Sample Size Formulation: $$n = \frac{2 \times (Z_{\alpha/2} + Z_{\beta})^2 \times p \times (1 - p)}{\delta^2}$$
  • $p$: Baseline conversion rate; $\delta$: Absolute detectable effect ($p \times \text{MDE}$).
• Runtime Constraint Rule: If your monthly traffic requires more than 4 weeks to reach statistical power, the test is underpowered; choose a higher-traffic surface or test a bolder structural variation.