Incident Response and Troubleshooting

Manage the incident (people, communication, control) and resolve it (mitigate first, then diagnose systematically).

Incident Response

Resolve the incident (mitigate impact, restore service) and manage it (coordinate people, keep everyone informed). Most incidents that spiral do so because of management failures, not technical ones.

The 3Cs

• Coordinate the response effort.
• Communicate among responders, to the organization, and to users.
• Control the response: one clear line of command; no freelancing.

Common failure modes to guard against: the on-call is so deep in the technical problem that nobody manages; nobody knows what others are doing; well-meaning people make uncoordinated changes that make things worse.

When to declare an incident

Declare early. It is cheap to close a small incident and expensive to organize a big one hours late. Declare if any is true: