tufte-data-forensics
Data Forensics: Auditing an Analysis Before You Trust It
Overview
Seeing With Fresh Eyes ch. 4 (printed pp. 80–119) is a procedure, not a lament. Its organizing device is a proposed reference work — an imagined Data Measurement and Analysis: 1,000 Safeguards, Pitfalls, and Cheats in Statistical Practice, modeled on Khonsari and Sintek's cardiac-surgery encyclopedia of some 3,000 graded alerts and warnings (p. 82). The parallel is the whole argument: surgery already has an itemized, severity-graded, actionable literature of things that go wrong at the point of work, and statistical practice does not — even though a botched analysis reaches more lives than a botched operation.
This skill owns the step-by-step audit. Two siblings own adjacent material and are not repeated here:
| Sibling | Owns |
|---|---|
tufte-evidence-corruption |
The catalog of corruption types — cherry-picking, effects without causes, model searching, overreaching, chartjunk, the rage to conclude, Chalmers, the "Which half?" exchange |
tufte-visual-thinking |
The seeing discipline and the ch. 4 highlights — Feynman/Mosteller, statistical vs. named lives, sampling-to-please, the Lucent triad, forensic-audit-vs-cleaning as named concepts |
What follows turns those named concepts into an executable audit: what to look at, in what order, what to demand, and what to write down.
The operating stance. Feynman's rule opens the chapter's facing page: the first principle is that you must not fool yourself, and you are the easiest person to fool (Feynman, quoted p. 80). Everything below is machinery for a person who has accepted that they are the primary threat. Tufte reinforces it with Darwin — that ignorance begets confidence more often than knowledge does (Darwin, quoted p. 86) — and with Van Wyck Brooks's principle that once you hold a point of view, all history will back you up (p. 81). The chapter's recurring villains are named as characters: Dr. Confirmation Bias and Dr. p-Hacker, who build narratives out of a fresh batch of random numbers (p. 86).
The selection-bias frame (p. 80). Tolstoy's remark that Napoleon "commanding" his armies compresses thousands of consecutive dependent commands, of which only the executed few are ever remembered, is the chapter's compact statement of causal selection bias: the record you audit has already been filtered by outcome before it reached you. Assume the survivors are in your dataset and the non-survivors are not.