ab-test-analysis

Installation
SKILL.md

Pre-checks (before reading results)

  1. Sample ratio mismatch (SRM) — the variant split should match the planned ratio (e.g. 50/50). If a chi-square test gives p < 0.001, treatment assignment is broken; do not trust the results.
  2. Coverage — every randomized unit should appear in the analysis. Missing units suggest log loss or a filter bug.
  3. Pre-period parity — variants should look similar on the metric before the test started. If not, the randomization is suspect.

Primary metric

  • Compute lift as (treatment - control) / control (relative) and treatment - control (absolute).
  • Report a confidence interval (or p-value) — point estimates without uncertainty are misleading.
  • Account for variance reduction if available (CUPED, regression adjustment) — usually halves the required sample size.

Guardrails

Even a winning treatment can be a net loss. Always check:

  • Latency / error rate
  • Bounce / churn / unsubscribe
  • Other key business metrics that shouldn't regress
Installs
2
GitHub Stars
7
First Seen
May 13, 2026