observability-rails-advanced
Installation
SKILL.md
Observability — Advanced
Beyond "we have logs + Sentry." This skill is about deciding WHAT to alert on, HOW LOUDLY, and ensuring on-call is woken only when a human can do something useful.
The opinion
Define SLOs first, alerts second. Use multi-window multi-burn-rate alerts (Google SRE Workbook) — alert when the error budget is burning fast OR slowly over a long window. Page only on customer-impacting events. Everything else is a ticket. Maintain runbooks for every alert. Use Prometheus + Grafana + Alertmanager as the open-source default; Datadog if you're willing to pay. Pair metrics with OTel exemplars so a Grafana panel can jump to a trace.
The four observability signals
| What | Default tool | |
|---|---|---|
| Logs | Discrete events, debug detail | structured JSON via lograge → Loki / Datadog Logs / Elastic |
| Metrics | Aggregated time series | Prometheus / Datadog |
| Traces | Cross-service request flow | OpenTelemetry → Tempo / Honeycomb / Datadog APM |
| Errors | Exceptions with context | Sentry / Rollbar / Datadog Errors |
See observability-baseline for setup; this skill is about strategy.