Writing runbooks that on-call engineers actually use

A runbook is the page an alert links to. It explains what the alert means, how to confirm it, what to do, and who to call if that fails. What to alert on lists the four things a minimal runbook needs. This guide covers how runbooks differ from playbooks and postmortems, a structure that works, how to link a runbook from a Prometheus or Grafana alert, how to keep runbooks current, a copy-paste template, and a worked example for a disk that is filling up. The Google SRE book reports that recording best practices ahead of time in a playbook gives roughly a 3x improvement in MTTR compared with improvising.

Runbook, playbook, postmortem

These terms overlap. Google's SRE book uses "playbook" for what most teams call a runbook. The kubernetes-mixin runbook page treats the two as the same thing. PagerDuty's runbook explainer separates them. A runbook covers one task. A playbook covers the response to a larger event and can contain several runbooks.

Document	Answers	Written	Example
Runbook	What do I do about this alert or task?	Before the alert fires, with the alert rule	"HostDiskFillingUp"
Playbook	How do we respond to this kind of event?	Before, for classes of incident	"Region outage", "Suspected data breach"
Postmortem	What happened, and what will change?	After an incident	" checkout errors"

A postmortem often produces a runbook change as one of its action items. See blameless postmortems.

What goes in a runbook

The kube-prometheus project publishes runbooks at runbooks.prometheus-operator.dev. Their structure relies on four headings: Meaning, Impact, Diagnosis and Mitigation. You should add an owner and an escalation path to cover most alerts.

• Meaning: State what the alert measures and why it fires. Keep it to one or two sentences.
• Impact: Describe what users or other systems notice if nobody acts.
• Diagnosis: Explain how to confirm the alert is real and find the cause. Include commands and dashboard links. Show what normal output looks like.
• Mitigation: List the steps to stop the impact. Start with the safest option. Mark any step that deletes data, restarts a service or needs approval.
• Escalation: Name who to bring in and when. Use this if mitigation does not work.
• Metadata: Record the owner, last reviewed date and a link to the alert rule.

Write for someone on call who has never touched this system. Spell out hostnames, namespaces and dashboards. Put commands in code blocks, ready to paste, with placeholders in angle brackets. Use one action per step, numbered. Say what to expect after each step so the reader knows whether it worked. Link the dashboard and the architecture doc instead of pasting screenshots that go stale.

Link it from the alert

The Prometheus ecosystem uses an annotation named runbook_url on each alerting rule. The Prometheus alerting rules docs say annotations hold longer descriptive information, such as runbook links. kube-prometheus sets runbook_url on every alert it ships, pointing to runbooks.prometheus-operator.dev (rule file). Grafana-managed alert rules have a built-in Runbook URL field. Grafana stores this as the runbook_url annotation and includes the link in notifications by default (Grafana docs).

An illustrative rule, checked with promtool check rules:

groups:
  - name: host-disk
    rules:
      - alert: HostDiskFillingUp
        expr: |
          node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 0.10
          and
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is predicted to fill within 4 hours"
          runbook_url: "https://wiki.example.com/runbooks/host-disk-filling-up"

The annotation only helps if it reaches the notification. In an Alertmanager template each alert's annotations are under .Annotations (notification template reference), so the text field of a Slack receiver can print the link for every alert in the group:

text: '{{ range .Alerts }}{{ .Labels.instance }} - runbook: {{ .Annotations.runbook_url }}{{ "\n" }}{{ end }}'

Put the link on every paging alert, and point it at a stable URL for that alert rather than a search page. For routing the notification itself, see Alertmanager routing and alert message design.

Keeping runbooks current

A wrong runbook is worse than none, because the reader trusts it. PagerDuty advises keeping runbooks specific to your own processes, field testing them and updating them regularly.

Practical habits (recommendations, not a standard):

• Fix it when you use it. The person who followed the runbook corrects anything wrong the same day, or files a ticket.
• Keep it next to the alert. Store runbooks in the same repository as the alert rules. A change to the rule and its runbook land in the same review.
• No runbook, no page. Do not ship a paging alert without a runbook_url. A CI check can grep rule files for it.
• Date it. Put an owner and a last reviewed date at the top. Set a review interval, for example six months.
• Delete with the alert. When an alert is removed, remove or archive its runbook.
• Make it a postmortem output. If responders had no runbook, or a wrong one, add "write or fix the runbook" as an action item.

Runbooks can also be partly or fully automated. PagerDuty describes three levels: manual, semi-automated and fully automated. The kubernetes-mixin page suggests automating many procedures instead of leaving them in a wiki. It notes that even a problem that fixes itself should come with an explanation of what happened.

A runbook template

Copy this into your wiki or repository, one file per alert:

# <AlertName>

Owner: <team or person>    Last reviewed: <YYYY-MM-DD>
Alert rule: <link to rule file or alert definition>
Dashboard: <link>

**Meaning**
<One or two sentences: what this alert measures and the condition that fires it.>

**Impact**
<What users or dependent systems notice if nobody acts, and how soon.>

**Diagnosis**
1. Confirm the alert is real: <command or dashboard>. Normal looks like: <expected output>.
2. Find the cause: <command>. Look for: <what a bad result looks like>.
3. Check recent changes: <deploy log, change calendar, config repo>.

**Mitigation**
1. <Safest step first>. Expected result: <what changes>.
2. <Next step>. Expected result: <what changes>.
3. RISKY: <step that deletes data, restarts or fails over>. Needs approval from: <role>.

**Escalation**
If not mitigated within <N> minutes, escalate to <team / rota> via <how>.
For customer impact, inform the incident commander so a status update goes out.

**Notes**
<Known false positives, related alerts, links to architecture docs and past postmortems.>

Example: a disk filling up on a Linux host

An illustration of the template filled in for the HostDiskFillingUp rule above. The commands are for a typical systemd-based Linux server; adjust paths to your hosts.

Meaning and impact

The alert fires when a filesystem has less than 10% free and the 6-hour trend predicts it will be full within 4 hours. When the disk fills, anything that writes to it fails, such as a database writing data or a service writing its logs. The prometheus-operator runbook for the similar NodeFilesystemSpaceFillingUp alert notes that performance usually degrades before the filesystem is completely full.

Diagnosis

1. Confirm free space and free inodes on the mount from the alert's mountpoint label. If IUse% is near 100%, the problem is too many files, not too much data.

df -h /var
df -i /var

2. Find the largest directories on that filesystem. -x stays on one filesystem; the last line is the total for /var itself. Repeat on the largest subdirectory until you find the cause.

sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail

3. Check the common culprits: the systemd journal, large log files, and files that were deleted but are still held open by a process (their space is not freed until the process closes them).

journalctl --disk-usage
sudo find /var/log -xdev -type f -size +100M -exec ls -lh {} +
sudo lsof +L1

Mitigation

1. Shrink the journal. --rotate archives the active files first, because --vacuum-size only removes archived journal files.

sudo journalctl --rotate --vacuum-size=500M

2. If one log file is huge and the service still has it open, empty it in place instead of deleting it, so the space is freed straight away. Only do this if the log is not needed for an investigation.

sudo truncate -s 0 /var/log/<service>/<file>.log

3. If lsof +L1 showed a large deleted file, restart the process holding it so the space is released.
4. On a Docker host, check what Docker is using before removing anything. docker image prune removes only dangling images unless you add -a.

docker system df
docker image prune

5. If the growth is application data, such as a database, do not delete files. Escalate to the owning team.

Escalation and follow-up

If free space keeps falling after these steps, escalate to the team that owns the host. Afterwards, fix the cause: set SystemMaxUse= in journald.conf, add log rotation for the file that grew, or resize the volume.

Tools

Some incident management and on-call tools attach runbooks to services or alerts, and some run automated steps. Browse the tools filtered by runbooks and the incident management directory.