Survivorship Bias

Overview

Abraham Wald (Statistical Research Group, WWII): commanders wanted to armor bombers where returning planes showed bullet holes. Wald inverted it — the returning planes survived those hits; armor where the survivors are clean, because planes hit there never came back. The holes you can see mark the survivable damage. The dataset "planes we can inspect" was silently filtered by the very outcome under study.

Every dataset arrived through a filter. Analyzing the data without modeling the filter analyzes the survivors, not the population.

When to Use

• User feedback / NPS / reviews (only non-churned, non-silent users respond)
• Error logs and crash reports (crashes that prevent reporting don't report)
• "What do successful X do?" pattern studies (failures did those things too — unobserved)
• Benchmarks, case studies, testimonials, A/B results, retention metrics
• Any conclusion of the form "our users want…" / "top performers all…"

The Procedure