A/B Testing Outreach

Most outbound "tests" aren't tests — they're a comparison of forty sends against forty other sends, a difference well within normal random variation, followed by a confident conclusion about which version "won." Running a genuinely valid test requires deciding on one variable, calculating a real sample size, and resisting the urge to call a winner before enough data exists.

Test one variable at a time

Weak     a new subject line, a new opening line and a new CTA all
         changed simultaneously between "version A" and "version B"
Strong   exactly one element changed, everything else held constant

Changing several things at once means any performance difference can't be attributed to any single change — if the "winning" version had a different subject line, opening and CTA, you've learned that some combination performed differently, not which specific change caused it, which makes the result nearly useless for informing the next test.

What's actually worth testing, roughly in order of impact