deciding-with-confidence
Deciding with confidence
scripts/decide.py takes an OpenAI Decisions API request (input plus
questions of type predicate, choice or score) and returns its response
shape: a probability per option, the chosen value or score, and confidence.
Claude Haiku 5.5 answers. Its logits are not exposed, so the distribution is
estimated: each of k parallel samples states a probability per option with the
options in a different order, the samples are averaged, and the average is
temperature-scaled from a labelled eval. Disagreement between samples lowers
confidence and shows up in diagnostics.agreement.
Expect about 85% of what a purpose-built decision model gets: on the bundled
104-item eval, accuracy 0.846 against Jev's 0.894, with a calibration error
(ECE 0.041) that matches it. The whole accuracy gap was on eight-way intent
questions; yes/no and rubric questions tied. Disagreement between samples is
the strongest warning sign. references/method.md has the numbers.