Survivorship Bias

Overview

Abraham Wald (Statistical Research Group memoranda, 1943; exposited in Mangel & Samaniego, JASA 1984): estimating aircraft vulnerability from returning planes is wrong unless you account for the planes that never came back — the returning planes survived their hits; the unhit areas on survivors mark where hits were fatal. (The vivid scene of commanders wanting to armor the bullet holes is a later retelling, via Wallis's 1980 memoir and Ellenberg — not in Wald's papers — but it dramatizes his math faithfully.) The dataset "planes we can inspect" was silently filtered by the very outcome under study.

Every dataset arrived through a filter. Analyzing the data without modeling the filter analyzes the survivors, not the population.

When to Use

• User feedback / NPS / reviews (only non-churned, non-silent users respond)
• Error logs and crash reports (crashes that prevent reporting don't report)
• "What do successful X do?" pattern studies (failures did those things too — unobserved)
• Benchmarks, case studies, testimonials, A/B results, retention metrics
• Any conclusion of the form "our users want…" / "top performers all…"

The Procedure