abtesting

abtesting

Purpose

This skill enables A/B testing workflows, including experiment design, statistical significance testing, sample size power analysis, and defining launch criteria. Use it to optimize decisions based on data-driven experiments.

When to Use

Apply this skill when designing experiments for feature rollouts, website changes, or product variations. Use it for scenarios requiring statistical validation, such as comparing conversion rates or user engagement metrics, to ensure reliable results before full deployment.

Key Capabilities

• Design A/B experiments with parameters like variants, metrics, and duration.
• Perform power analysis to calculate required sample sizes using formulas like Cohen's d.
• Test statistical significance with t-tests or chi-squared tests on experiment data.
• Define launch criteria based on p-values, confidence intervals, and effect sizes.
• Integrate with data sources for real-time analysis.

Usage Patterns

Start by initializing an experiment object with required parameters. Use CLI for quick calculations or API for programmatic access. Always set the API key via environment variable $ABTEST_API_KEY before operations. For example, chain commands to design, run analysis, and decide on launch. Handle asynchronous API calls by polling for results.