experimentation

Installation
SKILL.md

Experimentation

Most A/B testing programs produce confident conclusions from insufficient data. The discipline is almost entirely in what you do before launch.

Before running

  • Hypothesis with a mechanism. "Moving the pricing table above the fold will raise trial starts, because visitors currently leave before seeing pricing." Not "let's try a green button."
  • One primary metric, chosen in advance. Secondary metrics are context, never the verdict.
  • Sample size calculated in advance, from your baseline rate and the smallest lift that would change a decision. If the required sample is unreachable, do not run the test — decide by judgment and say so.
  • Duration set in advance, covering at least one full weekly cycle, and two if the buying cycle is long.
  • Guardrail metrics that would make you reject a win: refunds, support volume, downstream retention.

While running

Installs
4
GitHub Stars
1.3K
First Seen
12 days ago
experimentation — cbrock84/headcount