Confidence Calibration

Overview

Techniques for getting Claude to self-assess confidence on its outputs and using that signal to route decisions: auto-approve high-confidence answers, flag medium-confidence for review, and escalate low-confidence to humans. Critical for production systems where blind trust in model outputs creates risk.

The Confidence Scale

CONFIDENCE_PROMPT = """After answering, rate your confidence on this scale:

5 - Certain: The answer is directly supported by the provided context with no ambiguity
4 - High: Strong evidence supports the answer, minor interpretation needed
3 - Moderate: Some evidence supports the answer, but key details are missing or ambiguous
2 - Low: Limited evidence, significant assumptions required
1 - Very Low: Mostly guessing, insufficient information to answer reliably