Alerting

Every alert that fires and requires no action trains the person who gets paged to trust alerts a little less. That erosion compounds — a team that's been paged for three false alarms this month will be slower, not faster, to react to the fourth page, even if it's real. The job of an alerting system isn't maximizing coverage, it's maximizing the fraction of pages that were worth waking up for.

Coverage and trust trade off against each other more often than teams admit — an alert added "just in case" is rarely free, because it competes for attention with every alert that already exists.

An alert that doesn't require a human to do something right now shouldn't page a human right now.

1. Alert on symptoms, not on causes

A symptom is something the user is experiencing — elevated error rate, high latency, failed checkouts. A cause is an internal condition that might explain a symptom — a pod restarted, disk is at 82%, one replica is unhealthy behind a load balancer with four others. Alerting on causes pages people for things that often don't matter (a disk at 82% with an autoscaler about to add capacity) and misses things that do (five separate low-severity causes combining into a real user-facing outage).

• Symptoms page — error rate, latency, failed checkouts, budget burn.
• Causes inform the response, surfaced on a dashboard the paged engineer opens after the page, not baked into the page itself.
• A cause-based alert that's "usually fine" trains people to ignore it — which is exactly the erosion this skill exists to prevent.

Alert on the symptom; use the causes as the first thing you check once paged.

Done when: every paging alert is defined in terms a user of the system would recognize, not in terms of infrastructure state.