Reading list

Google's book on designing for security and reliability together, with chapters on crisis management, recovery and a resilient team culture.

Book-length guide to choosing SLIs, setting targets, running error budgets and getting an organisation to act on them, from a former Google SRE.

O'Reilly book on high-cardinality events, tracing and SLO-based alerting, and why debugging unknown failures needs more than dashboards. Honeycomb offers a copy in exchange for a form.

Short O'Reilly book on monitoring strategy: monitoring anti-patterns, what to alert on, on-call hygiene, and monitoring each layer from business metrics down to servers.

How Google runs production: SLOs, toil, monitoring, on-call, incident management and postmortems. The shared vocabulary most on-call and alerting practice now builds on.

Operations textbook for large web services with chapters on monitoring, alerting, on-call and learning from outages; broader than SRE but full of practical detail.

Hands-on companion to the SRE book with worked examples: implementing SLOs, alerting on burn rate, on-call case studies and running incident response.

How Atlassian detects, responds to and reviews incidents: roles, severity levels, communication and the postmortem process, written for others to adapt.

Introductory articles on on-call schedules, rotation types, pay and avoiding burnout.

Guide to facilitating post-incident debriefings that focus on learning rather than blame, with practical questions to ask responders.

GitLab's public handbook pages on its on-call processes and policies: rotation tiers, expectations, handover and holiday coverage. A real-world example to borrow from.

Google SRE guide covering preparation (symptom-based alerting, playbooks), incident command with IC, communications and operations leads, and blameless postmortems.

Step-by-step incident analysis guide originally written by Jeli: collecting data, interviewing responders, calibrating the review and sharing what was learned.

Vendor-written guide covering incident roles, declaring incidents, running the response, post-incident work and building an incident culture.

Training material for the incident commander role: taking command, running the call, delegating, and deciding when an incident is over.

PagerDuty's open incident response process: severity levels, incident commander and other roles, call etiquette, and what to do before, during and after an incident.

How to run blameless postmortems: who owns them, timelines, analysis, the review meeting, and following up on action items.

What an on-call engineer is expected to do: acknowledgement times, handoffs, when to escalate, and how to look after yourself during a rotation.

Ops guide for teams starting on-call: technical preparation, team norms, on-call culture, and recommendations for managers.

Walks through alerting designs from simple thresholds to multi-window, multi-burn-rate alerts on error budgets, with trade-offs for each.

How Google balances on-call load: time limits, incident counts per shift, compensation, and the resources an on-call engineer needs.

John Allspaw on why the search for a single root cause misleads incident reviews, and how contributing factors combine.

Eighteen short points on why failures in complex systems have no single root cause and why operators constantly prevent failure. Foundational for blameless reviews.

Incident command in practice with case studies from Google and PagerDuty, plus best practices for declaring and handing over incidents.

Increment's inaugural issue on on-call and incident response, with essays and interviews on how many companies run their rotations.

A badly and a well handled incident compared, and the roles (incident commander, ops, communications, planning) that make the difference.

Sorts alerts into pages, notifications and records by urgency, and argues for paging on symptoms rather than causes.

Defines symptom-based alerting and the four golden signals (latency, traffic, errors, saturation), and sets out what should page a human.

Classic essay arguing pages must be urgent, actionable and user-visible, and that cause-based alerts create noise. The seed of the SRE book monitoring chapter.

Charity Majors on why engineers should be on call for their code, and what managers owe them in return: fewer pages, time to fix things and real support.

Case studies of improving on-call at Google and elsewhere: reducing pages, rotation design, handoff and reducing toil for responders.

When to write a postmortem, keeping it blameless, and making sure reviews and action items actually happen.

Explains Tom Wilkie's RED method: track rate, errors and duration for every service, giving a consistent base for dashboards and alerts.

Checklist for performance and health checks: for every resource, look at utilisation, saturation and errors. Useful for deciding what host-level signals to watch.

Open YAML specification for defining SLOs, SLIs and alert policies in a vendor-neutral way.

The vendor-neutral spec for traces, metrics and logs, including APIs, SDKs, semantic conventions and the OTLP protocol.

Reference for Alertmanager's grouping, inhibition, silencing and routing trees, the building blocks many alert pipelines copy.

Prometheus docs page on keeping alerts simple, paging on symptoms, and alerting on user-visible latency and errors for online and batch services.

Postmortem template and field-by-field guidance from Atlassian's incident handbook.

A complete sample postmortem from the SRE book, with summary, impact, root causes, timeline and action items, ready to copy as a template.

Sample SLO document for a fictional service, with SLIs, targets, rationale and the error budget policy attached.

PagerDuty's postmortem template with sections for summary, impact, timeline, contributing factors and follow-up actions.

Runbooks for each alert in the kube-prometheus stack: what the alert means, its impact, how to diagnose and how to fix it. A model for writing your own.

Markdown template for service run books: architecture, monitoring and alerting, failover, backup and troubleshooting sections.

Google's free CC-BY workshop (slides, participant handbook, worksheet, facilitator notes) for teaching teams to define SLIs and SLOs for a fictional game.

SREcon18 Asia talk on paging only when the error budget is burning fast enough to need a human.

SREcon16 Europe talk on moving SoundCloud from noisy cause-based pages to symptom-based alerting with Prometheus.

SREcon19 EMEA talk on Zalando's adaptive paging, which uses tracing to page the team closest to the problem instead of every layer.

SREcon20 Americas talk on research into pager fatigue and the team practices that reduce it.

Collection of ready-made Prometheus alert rules for common databases, brokers, hosts and Kubernetes, with thresholds to adapt.

Curated list of monitoring, metrics, logging and alerting tools, mostly open source.

Curated list of observability tools and reading across metrics, logs, tracing and alerting.

Curated list of SRE reading: books, articles, talks and tools on monitoring, on-call, incident response and postmortems.

Curated list of hosted and open source status page services and tools.

Large list of software and services with free tiers for developers, including monitoring, log management and error tracking sections.

Links to public writing on how technology companies practise SRE, including on-call, incident management and SLOs.

Dan Luu's collection of public postmortems from many companies, grouped by type of failure. Good material for learning how outages actually happen.

Lorin Hochstein's annotated list of resilience engineering and safety science papers relevant to software incidents.

Searchable database of public incident reports with analysis of trends in how software organisations describe failures.