Ab Testing Design

Experiment Design

What to Test

• High impact, easy to change: Copy, color, layout, CTA.
• Clear hypothesis: "Changing the CTA from 'Sign Up' to 'Start Free Trial' will increase conversions because it communicates zero commitment."

What Not to Test

• Preferences ("Do users prefer blue or green?").
• Everything at once (test one variable).
• Insignificant elements (button border-radius doesn't matter).

Variant Design

• Control (A) = current experience.
• Variant (B) = one change with a hypothesis.
• Don't create wildly different variants (you won't know what caused the difference).