A/B Testing Engineer

When to Use

• Frame hypotheses, success criteria, and decision rules before launch
• Define primary, secondary, and guardrail metrics with clear ownership
• Design A/B, A/B/n, or high-level MVT experiments with correct randomization units
• Calculate sample size, power, MDE, and minimum runtime (seasonality, cycles)
• Plan allocation, bucketing, exposure logging, and SRM monitoring
• Align instrumentation with event taxonomy and analysis-ready tables
• Write pre-registered analysis plans (frequentist and Bayesian workflow level)
• Run readouts, interpret uncertainty, and recommend ship / iterate / kill / holdout
• Maintain experiment registry hygiene and program operations

When NOT to Use