A/B Testing & Experimentation Skill

How this skill behaves (read first)

This is a generative skill (it designs or revises an experiment plan, and can review an existing test setup). Experimentation is where an AI assistant's defaults are quietly wrong in ways that produce confident, false conclusions. Claude will happily "set up an A/B test" that declares a winner on a handful of users, peeks and stops the moment p < 0.05, optimizes one metric with no guardrail for the damage it does elsewhere, runs for three weekdays, and ends at "B won" with no segments, no long-term read, and nothing written down. A bad experiment is worse than none — it launders a guess into "data."

So this skill gates, and the gate is statistical discipline:

1. Establish context — is a controlled experiment even the right tool here, what single decision it informs, and the risk level (these set confidence, split, and duration).
2. Apply the always-true core — pre-register the hypothesis and success criteria, size the test before running it, run a full cycle, use one primary metric + guardrails, interpret past "it won," and close the learning loop.
3. Surface the context-dependent decisions (confidence threshold, traffic split, A/B vs. multivariate, metric type, leading/long-term indicators, test prioritization) with trade-offs; let the user choose.

Then it hands off to pm-okr-metric-validity-audit (are the chosen metrics valid, not vanity?) and pm-assumption-rigor-audit (is the hypothesis the riskiest, falsifiable, pre-committed?).

Scope & pairing: this skill owns the controlled-experiment method and its statistics. It pairs with pm-assumption-testing, which owns choosing what to test and the cheapest way to test it — when the question is early-stage, low-traffic, or better answered qualitatively, defer to assumption-testing rather than forcing an A/B test. It defers broad what-to-build-next ranking to pm-prioritization, metric definitions/OKRs to pm-okrs-kpis, and the high-stakes roll-out go/no-go to pm-decision-quality-audit.

Step 0 — Establish context before designing the test