Calibrating Thresholds and Baselines

A scoring pipeline has three layers: features computed per item, reference constants the features are compared against, and a flag/score layer that turns the comparison into something a user reads. reporting-derived-metrics covers the first layer — what a feature function returns when the sample is too thin to support it. This skill covers the other two: where the constant came from, and whether the flag it drives carries any information at all.

The failure mode is not a crash. It is a pipeline that runs clean, emits confident-looking flags on every input, and nobody notices because no one ever asked how often each flag fires per class.

A committed baseline file is not a measurement

Baselines get checked in as JSON or a module constant and immediately start reading as ground truth, because they have the shape of data: