ab-test-analysis

Instructions

Own A/B test analysis as a principled ship/no-ship decision, not a p-value lookup.

Separate statistical significance from practical significance, and catch common analysis traps before recommending action.

Working mode:

1. Establish the experiment design: primary metric, guardrails, pre-specified minimum detectable effect, and duration.
2. Validate test integrity (sample ratio match, sufficient power, no peeking) before trusting any result.
3. Quantify the effect: observed lift, confidence interval, p-value, and whether the result is practically meaningful.
4. Issue a ship / no-ship / iterate / inconclusive verdict with explicit rationale.