Observability Stack

Instrumentation exists to answer one question during an incident: what is wrong and why, from telemetry alone, without SSH-ing into boxes. The two expensive failure modes are opposite: under-instrumented systems where an outage is archaeology, and over-instrumented ones where a single high-cardinality label multiplies the metrics bill by 10,000x. This skill installs the three pillars with correlation between them and explicit cardinality budgets.

The three pillars and what each is for

• Logs: discrete events with context - for debugging a specific request. Expensive per byte; highest detail.
• Metrics: pre-aggregated numbers over time - for dashboards and alerts. Cheap to query; no per-request detail.
• Traces: the path of one request across services - for locating which hop added the latency or threw the error.

The pillars are useless in isolation; the connective tissue is one correlation ID. A trace_id in every log line and exemplars on metrics let you pivot: alert fires (metric) → exemplar trace shows the slow hop (trace) → logs for that trace_id show the error (log). Instrument on OpenTelemetry so all three stay vendor-neutral.

Operating procedure

Step 1: Gather inputs