trustworthy-experiments
Trustworthy Experiments
What It Is
Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.
The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"
Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:
- 66-92% of experiments fail to improve the target metric
- 8% of experiments have invalid results due to sample ratio mismatch alone
- When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk
This framework helps you avoid the common traps that make experiment results untrustworthy.
Response Posture
More from wdavidturner/product-skills
strategic-narrative
Use when asked to "strategic narrative", "Andy Raskin", "tell our company story", "write a pitch deck", "explain why customers should care", or "movement narrative". Helps craft compelling narratives that define movements rather than just selling products. The Strategic Narrative framework (created by Andy Raskin) transforms pitches from feature lists into stories about change.
34thinking-in-bets
Use when asked to "thinking in bets", "make decisions under uncertainty", "think probabilistically", "avoid resulting", "separate decision quality from outcomes", or "reduce bias in decisions". Helps make explicit bets and evaluate decisions on process, not results. The Thinking in Bets framework (from Annie Duke) applies poker strategy to business and life decisions.
32okrs
Use when asked to "set OKRs", "objectives and key results", "quarterly OKR planning", "align objectives", "measure OKR progress", or "focus priorities with OKRs". Helps teams focus on what matters most and create a cadence of progress. The OKR framework (originated by Andy Grove at Intel, popularized by John Doerr at Google) creates alignment, focus, and learning cycles. Christina Wodtke's Radical Focus approach emphasizes simplicity and avoiding common pitfalls.
31design-sprint
Use when asked to "run a design sprint", "5-day sprint", "prototype in a week", "test ideas before building", or "Jake Knapp sprint". Helps teams go from problem to tested prototype in five days. The Design Sprint framework (created by Jake Knapp at Google Ventures) compresses months of work into one focused week.
26hierarchy-of-engagement
Use when asked to "define our core action", "North Star metric", "accruing benefits", "improve retention mechanics", "hierarchy of engagement", or "Sarah Tavel framework". Helps consumer products identify the actions and benefits that drive long-term retention. The Hierarchy of Engagement framework (created by Sarah Tavel at Benchmark) maps progression from core action to mounting loss.
26hooked-model
Use when asked to "build habit-forming products", "Hooked model", "trigger action reward investment", "create sticky behavior loops", or "design habit loops". Helps design products that form unprompted user habits. The Hooked Model (created by Nir Eyal) explains how products create habits through Trigger, Action, Variable Reward, and Investment.
25