experiments-and-holdouts

Two questions get confused constantly, and the tools for them are different.

"Which version is better" is a question for an A/B test: both groups receive something, one element differs. "Does this do anything at all" is a question for a control group: one group receives nothing. You can win the first and still have no idea about the second, which is how programs end up with a folder of winning tests and flat revenue.

This skill covers designing one comparison, building the control group it needs, and reading what comes back.

When to use this