abtesting-stats

Purpose

This skill performs statistical analysis for A/B testing, including t-tests, chi-squared tests, Mann-Whitney U tests, p-value calculations, confidence intervals (CIs), multiple testing corrections like Bonferroni or Benjamini-Hochberg (BH), and Bayesian A/B testing methods.

When to Use

Use this skill when comparing two groups in experiments, such as website variants, to determine statistical significance. Apply it for hypothesis testing in data science workflows, validating A/B test results, or analyzing user behavior metrics. Avoid if data is non-numeric or sample sizes are too small (<10 per group).

Key Capabilities

• Conduct t-tests for normally distributed data.
• Perform chi-squared tests for categorical data.
• Run Mann-Whitney U tests for non-parametric comparisons.
• Calculate p-values and 95% CIs for effect sizes.
• Apply corrections like Bonferroni for multiple comparisons or BH for FDR control.
• Execute Bayesian A/B tests using priors like uniform or beta distributions.
• Handle input data from CSV, JSON, or in-memory arrays.

Usage Patterns

Invoke via CLI for quick runs or integrate via API for scripted workflows. Always provide data sources and specify the test type. Use JSON config files for complex parameters. For example, pipe data directly into CLI or call API endpoints in loops for batch processing. Ensure data is pre-cleaned (e.g., remove NaNs) before use.