Monitoring and Alerting

Monitoring answers what is broken and why. Page humans only when the error budget is under real threat; everything else is a ticket or a dashboard.

Monitoring Design

Monitoring exists to answer two questions: what is broken (symptom) and why (cause). Build it so that a human is only interrupted for the first, and can find the second quickly.

Principles

• Alert on symptoms, investigate with causes. Users feel "checkout is slow", not "DB CPU is 90%". Page on user-visible symptoms (ideally SLO burn — see Alerting on SLOs below); keep cause metrics for dashboards and debugging.
• Every page must be urgent, actionable, and need human judgment. If a script could respond, automate it. If it can wait until morning, it's a ticket. If nobody acts on it, delete it.
• Keep it simple. Rules that rarely fire, signals nobody looks at, and clever heuristics rot. Prefer a small set of well-understood signals and remove what isn't used (review quarterly).
• Black-box + white-box. Black-box probes show what users see right now (and catch problems that never reach your servers). White-box metrics show internals and imminent problems. You need both.
• Measure distributions, not averages. Use histograms/percentiles for latency; averages hide the tail. Choose bucket boundaries that match SLO thresholds.

The four golden signals (request-serving systems)