Calibration

Overview

Lichtenstein, Fischhoff & Phillips (1982) and decades of calibration studies: when people say they're 99% sure, they're right roughly 80% of the time; "100% certain" answers are wrong 10–20% of the time. Alpert & Raiffa (1982): 98% confidence intervals capture the true value ~60% of the time — people's ranges are 2–3× too narrow. Confidence is a feeling generated by fluency and coherence of the story in your head (Kahneman: "confidence is a feeling, not a judgment"), and it tracks story quality, not truth.

The blind spot: certainty is experienced as a property of the world. It's a property of your narrative.

When to Use

• About to say "definitely", "certainly", "I'm 100% sure", "this can't fail"
• Producing ranges/intervals (estimates, forecasts, error bars)
• High confidence formed quickly, from fluent/familiar material
• Evaluating whether to double-check something that "doesn't need it"

The Procedure