Ab Testing

Scope Boundaries

• Use when product, pricing, ranking, recommendation, or policy changes may impact user/business KPIs and causal validation is required.
• Use proactively when metric impact is uncertain, stakeholder opinions conflict, or ship/no-ship criteria are not explicit.
• Use when canary metrics alone are insufficient to support a decision.
• Do not use for deterministic functional verification; use testing-*.
• Do not use for long-term reliability telemetry design; use observability-*.

Goal

Produce causally valid, operationally safe, and decision-ready experiment outcomes.