Impact report

Answer "did it help?" with numbers a skeptic accepts, then show them as dark chart panels that read at Slack size. Scripts are in scripts/. examples/blacksmith-4vcpu.json is a complete report: the CI runner upgrade of Oct 2026, posted to #engineering.

1. Frame the comparison

Name the change, when it landed (merge or deploy time), the metric, and which direction is better. Then fix four things:

• Baseline: the closest comparable window before the change. A same-day baseline beats a whole week: CI medians drifted from 7.4 to 11.7 min across one week with no related change. Show the wider window only as context.
• Classification: decide before or after per sample by what actually ran (runner label, version), with the clock as fallback. Branches opened before the merge keep running the old setup.
• Control: something in the same windows the change cannot touch, such as jobs left on the old runner or an unchanged endpoint. A flat control is what rules out "it was just a quieter afternoon".
• Outcome: the number people feel (time until a PR is all green, end-to-end latency) next to the part you changed. When the part improves more than the outcome, name the new bottleneck.

For GitHub Actions timings, follow references/github-actions.md; its scripts fetch the data and compute all four.

Done when every headline number has a baseline, an after, n on both sides, and a control.

2. Compute