A/B Test Readout Skill

The hard part of an experiment is the readout: not "B won" but "is this real, is it big enough to matter, and should we ship?" This skill turns results into an honest decision — and flags the ways A/B results lie.

Working from a brief

Given results (even partial), write the full readout anyway. If significance isn't provided, reason about it from the numbers and flag what's needed to confirm. Mark assumed figures. Never declare a winner without addressing significance and sample.

Required Inputs

Ask for (if not already provided):

• The hypothesis and the primary metric
• Results — control vs variant: conversions/rate, sample size per arm, duration
• Guardrail metrics (revenue, retention, latency, complaints) that mustn't regress
• Pre-registered decision rule (what would count as a win) if one exists

Output Format