AI SRE Incident Response

Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.

When to Use This Skill

• An LLM endpoint begins returning degraded or hallucinated answers
• Token spend spikes beyond budget thresholds
• A model provider goes down and traffic must fail over
• Safety guardrails fire at abnormal rates
• A new model deployment causes latency or accuracy regression

Prerequisites

• Prometheus and Alertmanager deployed with scrape targets for AI services
• Grafana dashboards for golden signals (latency, error rate, cost, quality)
• On-call rotation configured in PagerDuty, Opsgenie, or equivalent
• Runbook repository accessible to responders
• Rollback mechanism for model and prompt versions (GitOps or feature flags)