A/B Test Design & Significance

Tell the user whether a test result is real or noise, and how big a test needs to be before it can answer the question. Most teams call tests too early on too little data; this keeps the decision honest with a proper two-proportion z-test.

When to use

The user has an A/B test result to evaluate, or wants to size a test before launching it.

Workflow

To evaluate a result: gather visitors and conversions for control and variant, pick a confidence level (usually 95%), and run the engine. To plan a test: take the baseline conversion rate and the minimum lift worth detecting (the MDE), and compute the required sample size.

Paths assume you installed with npx skills add. From a clone of this repo, use skills/ab-test-significance/resources/… instead.