Incident response basics

Escalation ends when someone acknowledges. Incident response covers everything after: diagnosing the issue, involving the right people, keeping stakeholders informed, and learning from the event. Minor issues fixed quickly need no process. Major incidents do. Agree on the process before the crisis hits.

Declare the incident

Declaring an incident means stating in a known place that this is an incident, its severity, and who leads it. Common triggers include:

• The problem meets a severity level that requires it.
• More than one person is needed.
• Customers are affected, or will notice.
• You have worked on it alone for a set time, such as 30 minutes, without progress.

Declaring early is cheap. An incident can be downgraded or closed in a minute.

Roles

For larger incidents, split the work. Role names vary; these follow the PagerDuty Incident Response documentation, which is published openly and widely copied.

• Incident commander: coordinates. Decides who does what, when to escalate, and when the incident is over. Usually does not debug.
• Subject matter experts: investigate and fix, reporting to the commander.
• Scribe: records the timeline: what was seen, what was tried, what was decided and when.
• Communications lead: updates the status page, support teams and stakeholders, so the people fixing it are not interrupted.

When only two or three people respond, one person may hold several roles. Hold the commander role separately as soon as more than two people are involved.

Communicate

• One place to talk. Use a dedicated chat channel or call for each incident. Many incident tools create one automatically.
• Regular updates. Send a short update at a fixed interval, even if nothing changed. "Still investigating, next update in 30 minutes" stops people asking.
• A status page for users. Say what is affected, not the internal cause. Post when it starts, when there is progress and when it is resolved.

Resolve, then follow up

An incident is resolved when users are no longer affected, even if the underlying cause is not yet fixed. Record the follow-up work as tickets before closing.

The postmortem

A postmortem, or incident review, is a written account of what happened and what will change. For serious incidents it should be required, not optional. A useful one has:

• A summary and the user impact, with times.
• A timeline, largely from the scribe's notes.
• Contributing factors: the conditions that made the failure possible.
• What went well, as well as what did not.
• Action items, each with an owner and a date.

Keep it blameless: the aim is to understand how the system, including its tools and processes, allowed a reasonable person's action to cause harm. If people expect blame, they stop telling you what happened. Postmortem templates are in the reading list.

Tools

Use incident management tools to handle channels, roles, timelines, and postmortems. See the incident management directory. Use status pages to communicate with users. See the status pages directory.