Groq Incident Runbook

Overview

Rapid incident response procedures for Groq API failures. Groq is a third-party inference provider -- when it goes down, your mitigation options are: wait, fall back to a different model, or fall back to a different provider.

This SKILL.md is the high-level flow. Deep, copy-paste-ready material lives in references/:

• triage-and-diagnostics.md — severity table, the Quick Triage script, and the full decision tree.
• mitigations.md — fallback-model routing (TypeScript), 429 rate-limit actions, 401 key rotation.
• communication-and-postmortem.md — Slack/status-page templates, evidence collection, postmortem template.

Prerequisites

• GROQ_API_KEY exported in the environment you run the triage commands from.
• curl for API probes; kubectl only if you collect logs from a Kubernetes deployment.
• Access to console.groq.com to rotate keys or upgrade the plan.
• A configured fallback provider (e.g. OpenAI) if you need to fail away from Groq entirely.

Authentication: every Groq API call in this runbook authenticates with a bearer token — Authorization: Bearer $GROQ_API_KEY. Keep the key in a secret manager, never inline; the evidence-collection step in communication-and-postmortem.md redacts gsk_ tokens from logs before archiving.