Experimentation

Purpose

Design and analyze rigorous A/B tests and experiments. Enforce proper sample size planning, pre-registration, statistical methods, guardrail monitoring, and decision frameworks.

Agent Protocol

Trigger

Exact user phrases: "A/B test", "experiment", "randomized trial", "sample size", "power analysis", "hypothesis test", "p-value", "multiple testing", "guardrail metric", "experiment design", "treatment effect", "H0", "alternative hypothesis", "statistical significance", "practical significance", "MDE", "minimum detectable effect".

Input Context

• Experiment type (A/B, multivariate, MAB, switchback)
• Primary metric(s) and their baseline values
• Minimum detectable effect (MDE) in absolute or relative terms
• Traffic volume and expected experiment duration
• Unit of randomization (user, session, cluster)
• Guardrail metrics and acceptable thresholds
• Regulatory and ethical considerations
• Existing experiment platform and tooling