Experimentation

Concept of the skill

An experiment converts a belief that a change is better into evidence that it is. The discipline is not "run an A/B test and read the p-value" — it is designing the comparison so the measured difference is caused by the change and nothing else, sizing it so a real effect is detectable above noise, picking metrics that capture the decision you actually face, watching the integrity diagnostics that catch a broken pipeline, and reading the result without fooling yourself. Random assignment is the engine: it makes treatment and control exchangeable, so any difference between them can only have come from the treatment — the counterfactual of what would have happened without the change.

Most failed experiments fail before analysis. The unit was wrong, the metric was a proxy trap, the test was too small, users interfered with each other, a feature flag assigned users inconsistently, the result was peeked at until it looked significant, the treatment was diluted by users who never saw it, or a sample-ratio mismatch made every metric suspect. The agent's job is to protect the causal claim and the decision, not to decorate an already-made choice with statistics.

Coverage