Calibration

Overview

Lichtenstein, Fischhoff & Phillips (1982; figures from Fischhoff, Slovic & Lichtenstein 1977) and decades of calibration studies: when people put the odds at 100:1, they're right ~73% of the time; "100% certain" answers are wrong 15–30% of the time. Alpert & Raiffa (1982): 98% confidence intervals capture the true value only ~60% of the time, and 50% interquartile ranges only ~33% — intervals would need to be roughly 2–3× wider (implied by the coverage gap, not reported as such). Confidence is a feeling generated by fluency and coherence of the story in your head (Kahneman: "confidence is a feeling, not a judgment"), and it tracks story quality, not truth.

The blind spot: certainty is experienced as a property of the world. It's a property of your narrative.

When to Use

• About to say "definitely", "certainly", "I'm 100% sure", "this can't fail"
• Producing ranges/intervals (estimates, forecasts, error bars)
• High confidence formed quickly, from fluent/familiar material
• Evaluating whether to double-check something that "doesn't need it"

The Procedure