Go A/B benchmarking

An A/B delta is valid only when arm identity is decoupled from time, work, and construction. p=0.000 asserts none of that — it only asserts within-run consistency, which a systematic per-arm bias satisfies perfectly. Use controlled arm order and inspect the raw samples before attributing a change.

Interleave inside the parent

testing walks a parent benchmark once to discover its b.Run leaves, then repeats each leaf -count times back-to-back. Five arms under -count=20 yield five contiguous blocks of twenty — never an interleaving — so monotonic thermal/frequency drift is fully confounded with arm identity. Shuffling arm order inside the parent does not fix it: the parent body runs once, so the shuffle only picks which arm owns which block.

Do the repetition inside the parent, at -count=1:

for rep := range reps {
    rand.Shuffle(len(arms), func(i, j int) { arms[i], arms[j] = arms[j], arms[i] })
    for _, arm := range arms {
        b.Run(fmt.Sprintf("rep=%d/arm=%s", rep, arm.name), arm.fn)
    }
}