conclusive-failure-experimentation
The goal of this skill is to ensure that when a product experiment fails, it does so "conclusively." By maximizing the treatment effect, you eliminate the ambiguity of whether an idea failed due to a poor hypothesis or just poor execution, allowing the team to move on or iterate with confidence.
The Conclusive Design Principles
In environments where you lack massive scale (common in B2B or early-stage startups), you cannot rely on a large sample size ($N$) to detect small improvements. Instead, you must maximize the Treatment Effect.
1. Maximize the Treatment Effect
If you have a hypothesis, do not test the "minimum" version. Throw every possible tactic and resource at the experiment to give it the best possible chance of success.
- Rationale: If it fails in its most "expensive," high-effort, and polished version, you can be certain the hypothesis is wrong.
- Action: If it works, you can "cost-rationalize" later by stripping away the parts that didn't matter.
2. Solve for the "Zombie Idea"
A "Zombie Idea" is a project that fails, is killed, but then is resurrected a year later because "maybe we didn't do it right last time."
- Action: Design the test so that you can tell future stakeholders: "We tried every lever (design, copy, trigger, and personalization) simultaneously, and it still didn't move the needle."