ab-test-analysis
Installation
SKILL.md
Pre-checks (before reading results)
- Sample ratio mismatch (SRM) — the variant split should match the planned ratio (e.g. 50/50). If a chi-square test gives p < 0.001, treatment assignment is broken; do not trust the results.
- Coverage — every randomized unit should appear in the analysis. Missing units suggest log loss or a filter bug.
- Pre-period parity — variants should look similar on the metric before the test started. If not, the randomization is suspect.
Primary metric
- Compute lift as
(treatment - control) / control(relative) andtreatment - control(absolute). - Report a confidence interval (or p-value) — point estimates without uncertainty are misleading.
- Account for variance reduction if available (CUPED, regression adjustment) — usually halves the required sample size.
Guardrails
Even a winning treatment can be a net loss. Always check:
- Latency / error rate
- Bounce / churn / unsubscribe
- Other key business metrics that shouldn't regress