Incident response

An incident issue is not a bug report. The goal is to stop the bleeding first and explain it second, and those two are frequently in tension.

Establish, in this order

1. Blast radius. Who is affected and how badly. An error rate that doubled from 0.01% is not the same incident as one that doubled from 4%.
2. When it started. Get a timestamp before you get a theory. The first timestamp people offer is usually when somebody noticed, not when it began.
3. What changed. Call correlate_deploys with the service and a window that starts before the timestamp from step 2. An empty result is informative: it means this is probably not a deploy, and you should stop looking there.

Do not skip to step 3. A deploy that landed in the window is not automatically the cause, and the fastest way to waste an hour is to roll back the first plausible change and watch the incident continue.