alert-routes
Installation
SKILL.md
Alert routes
Every alert rule in observability/prometheus/rules/ carries three things:
| Field | Where | Meaning |
|---|---|---|
route |
label | The team that owns the alert. Alertmanager routes on it. |
severity |
label | page: act now, at any hour. ticket: next working day. info: no notification, visible in Grafana and Alertmanager. |
runbook_url |
annotation | A runbook page in this repository (path from the repository root) or an http(s) URL. |
observability/alertmanager/alertmanager.yml has one child route per team, matched on route, and below it one route per severity. Each leaf has its own receiver, named <team>-<severity>. Alerts with no route label, or a value that no route matches, go to the receiver default. The gate check observability.alert-routes enforces the labels and the runbooks on every run.
Inputs
- The teams: short names, and for each team the person who keeps its routes and runbooks current.
- Per team and severity: the integration the receiver uses (webhook, chat, email, paging service) and where its secret lives. The user creates the secret files; never ask for the secret values in the chat.
- The alert rules to add or change, if any: the symptom, the query, the threshold and the team.