Observability

Monitoring answers questions you thought to ask in advance: is CPU high, is the queue backing up. Observability answers questions you didn't think to ask until the incident was already happening — why did this customer's this request fail on this pod during this deploy. You cannot dashboard your way to that; you need enough structured, correlated context emitted at the time the event happened that you can slice it afterward in ways you never pre-built a panel for.

That distinction should drive every instrumentation decision: not "does this look good on a dashboard" but "if this broke at 3am, could I find the cause from what I emitted." Instrument for the incident you haven't had yet, not the one you just fixed.

1. Pick the signal that matches the question

Metrics, logs, and traces are not interchangeable — each is cheap for a different question and expensive for the others:

A common mistake is logging what should be a metric (counting errors by parsing log lines) or trying to alert on high-cardinality logs instead of a counter. Decide the signal type before you write the instrumentation, not after storage costs surprise you.