Observability Stack Setup Skill

Goal

Produce a concrete observability setup plan that instruments a system across the three pillars (distributed traces, metrics, structured logs) using OpenTelemetry (OTel), correlates them, and defines the SLO/SLI targets and alerts that turn telemetry into action. For systems that call LLMs, the plan must emit spans that follow the OTel GenAI semantic conventions (gen_ai.* attributes) so model behavior, token usage, and cost are first-class signals. The output is a plan, not running infrastructure; humans review and apply it.

Steps

1. Inventory the system. List services, async boundaries (queues, event buses, background workers), external dependencies, and existing telemetry. Identify which services already export OTel data and which are blind spots. Note GABBE's own gabbe/audit.py already emits OTel GenAI spans with canonical gen_ai.* attributes — reuse it as the reference pattern rather than re-instrumenting LLM calls from scratch.
2. Stand up the three pillars. For each service specify: (a) Traces via OTel SDK auto + manual instrumentation; (b) Metrics via OTel meters (RED: Rate, Errors, Duration; plus saturation); (c) Logs as structured records correlated to spans. Route all three through an OTel Collector to your backend(s) so the SDK stays vendor-neutral.
3. Instrument LLM calls with GenAI conventions. For every model invocation, create a span named gen_ai.<operation> and set: gen_ai.system (provider), gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.usage.total_tokens. Derive a gen_ai.cost style attribute (or metric) from token counts × the per-model rate. Keep prompts/completions out of span attributes unless content capture is explicitly approved (see Guardrails).
4. Wire exemplars. Attach trace exemplars to latency and error histograms so a spike on a dashboard links directly to a representative trace. This is the bridge from "the p99 is bad" to "here is the slow request."
5. Propagate trace_id across async services. Inject and extract W3C traceparent context through every async hop — HTTP headers, message metadata on queue producers/consumers, and scheduled jobs. A dropped context breaks the trace; verify continuity end-to-end across at least one async path before declaring done.
6. Define SLIs and SLOs. Pick user-facing SLIs (availability, latency, error rate, and for LLM paths: successful-completion rate, token-budget adherence). Set SLO targets and the measurement window, and derive the error budget (1 - SLO). State which SLI each alert defends.
7. Bootstrap dashboards and alerts. Specify one overview dashboard per service (RED + saturation + GenAI token/cost panels) and alert rules that fire on SLO-burn symptoms (multi-window burn-rate), not raw causes. Every alert must name the runbook that handles it (see runbook-authoring.skill).
8. Write the plan. Assemble steps 1-7 into the Output Format below and hand it to a human for review.