Search procedures: how the garden actually gets walked

Why the procedure matters

Exhaustive enumeration is what a multiverse analysis does. It is not what a p-hacker does, and it is not what an agent under significance pressure does. They walk sequentially, with a stopping rule, along a path shaped by which knobs are easiest to turn. Two consequences:

• The false-positive rate is a property of the procedure, not of the grid. A grid of 25,920 specifications has a fixed min-p distribution; a modest hacker who stops at the first p < .05 after trying the clustering level, then the controls, then the window, has a different — usually lower — rate, and a different distribution of reported estimates.
• The honest p-value of a reported result is what that procedure would report on null data. Calibrating a sequential search against the exhaustive min-p distribution over-corrects; calibrating it against nothing under-corrects. search.null_calibration(procedure=...) replays the procedure itself.