Experiment Design

An experiment design document defines all parameters needed to run a rigorous A/B test or controlled experiment. It ensures the team aligns on what you're testing, how you'll measure success, and how long to run the test before drawing conclusions. Good experiment design prevents common pitfalls: underpowered tests, unclear success criteria, and decisions based on noise rather than signal.

When to Use

• Before launching an A/B test to validate a product change
• When testing a hypothesis that requires quantitative validation
• After solution design to validate assumptions before full rollout
• When stakeholders want data-driven evidence for a decision
• To establish a culture of experimentation and learning

When NOT to Use

• The hypothesis itself is not yet articulated -> use define-hypothesis first; this skill designs the test for a claim you already have
• You are analyzing a completed experiment -> use measure-experiment-results
• You need the event tracking that will measure the experiment -> use measure-instrumentation-spec
• You are gathering opinions rather than running a controlled test -> use measure-survey-analysis