performance-benchmarking

Producing performance numbers that mean something. The organizing fact is that a benchmark comparing identical code against itself can report a confident-looking improvement; every rule here follows from taking that seriously.

Vocabulary

Noise floor: the magnitude of difference a benchmark reports when nothing has changed. A property of the harness, machine, and workload, not of the code, and the threshold below which any measured effect is unmeasured rather than absent.

Point estimate vs interval: the point estimate is the headline; the interval is the part that says whether the headline means anything. Reading one without the other discards exactly the information that distinguishes a result from noise.

Paired measurement: measuring both arms under the same conditions, so shared variance (machine load, cache state, scheduler behavior) cancels in the difference instead of accumulating in it.

Gate vs signal: a gate blocks a merge under an agreed performance budget and measurement policy; a signal informs a reviewer. Use advisory results by default unless the project explicitly requires a performance gate.