Telemetry Operations

Use this skill to operate the telemetry stack — Prometheus, the OpenTelemetry Collector, and Loki — as the one deployment unit it ships as: collection and scraping, ingestion, retention, and the rules that turn raw signals into alerts. This is a tool skill for one stack of named tools. Observability strategy — SLIs, SLOs, error budgets, and what to instrument — belongs to platform-engineering; dashboards, panels, and Grafana-side alert rules, contact points, and notification policies belong to grafana. This skill owns the collection/ingest/retention layer and the Prometheus rules files that both of those skills consume.

Operating contract

1. Read-only discovery before any mutation. Inspect scrape configs, rules files, collector pipelines, and retention settings first. The bundled telemetry-check script runs rule sanity and scrape-target reachability checks without changing anything.
2. Confirm the target, scope, and rollback path before acting. Read-only discovery may proceed without confirmation. Mutations — a config reload, a promtool rules push, a collector restart, a retention-policy change — require an explicit human directive naming the instance.
3. A config that parses is not a config that works. Rule sanity catches structure; it does not prove the expression is meaningful or that the target is scrapable. Verify at the delivery boundary (scrape succeeded, rule evaluated, alert fired) before claiming health.
4. Keep evidence bounded. Summarize config diffs and query results; never dump full prometheus.yml, collector pipelines, or credentials into chat.
5. Own the retention decision. Retention is a capacity and compliance decision made deliberately per component — Prometheus block retention, OTel exporter buffering, Loki retention per tenant — and reviewed on a schedule, not left at defaults.

The telemetry-check script

scripts/telemetry-check is an agent-first, read-only checker. It parses Prometheus rules files with a bundled stdlib YAML reader and runs dependency-free structural sanity checks; it extracts static targets from scrape configs and probes TCP reachability; and it emits bounded JSON. It never writes files and never sends data anywhere.