loom-logging-observability
Installation
SKILL.md
Logging and Observability
Overview
Understand system behavior through the three pillars — logs, metrics, traces — correlated by shared IDs. This skill covers structured logging, OpenTelemetry tracing, Prometheus metrics, aggregation backends, and alerting, with emphasis on the cost/cardinality traps and sampling decisions that separate a working setup from an expensive broken one.
Three Pillars — what each answers, and its cost model
| Pillar | Answers | Cost driver | Use for |
|---|---|---|---|
| Metrics | "Is it broken? how much?" (aggregate) | Label cardinality (# series) | Dashboards, SLOs, alerting — always-on, cheap |
| Traces | "Where in the request path?" (causal) | Span volume → sampling | Latency breakdown, cross-service dependency |
| Logs | "What exactly happened?" (event detail) | Volume + indexing strategy | Forensics, audit, the specifics of one request |
Reach for metrics first (cheap, aggregate), traces to localize, logs for the detail. Link all three by trace_id/correlation_id so you can pivot: alert fires on a metric → jump to an exemplar trace → read that trace's logs.
Structured Logging
Emit JSON, one object per event — never string-interpolated prose. Structured fields are queryable in any backend; f"user {id} did {action}" is not.