Metrics and Monitoring

A metric is a compressed, aggregatable summary of something that happened many times — it deliberately throws away per-event detail to stay cheap at scale. That trade only works if you pick the right shape (counter, gauge, histogram) and keep the label set small; get either wrong and you've built either a metric that can't answer the question or a metrics backend that falls over under its own cardinality.

Prometheus's pull-based model made this trade-off explicit for a generation of tooling: a metric is a name plus a set of key-value labels, sampled on a schedule, and every design decision below follows from taking that model seriously rather than fighting it.

A metric earns its cost by staying cheap to query at 3am, six months from now, with a year of history behind it.

For PromQL patterns — rate, histogram quantiles, RED/USE queries, and recording rules — read references/promql.md.

1. Pick the metric type the query needs, not the one that's easiest to emit

The three core types answer different questions and are not interchangeable after the fact:

• Counter — a value that only goes up (requests served, errors, bytes sent). Never set it directly; use rate() over a window to get a meaningful number. Good for "how much happened."
• Gauge — a value that goes up and down (queue depth, in-flight requests, memory used). Good for "what is the state right now."
• Histogram (or summary) — a distribution bucketed by value, almost always used for latency. Good for "what's the p99," which an average can never tell you because it hides the tail.

Emitting latency as a gauge of the last request's duration, or errors as a gauge instead of a counter, are the two most common type mistakes — both make the eventual query impossible without redeploying the instrumentation.