What to alert on

Monitoring measures thousands of signals. Only a few warrant a page. The rest belong on dashboards, in tickets, or in logs for later search. This guide covers what to alert on and how to choose the right signals.

Page on symptoms, not causes

A symptom is what users experience. Requests fail. Pages load slowly. A job does not run. A cause is a reason a symptom might happen: high CPU, a full queue, a replica down.

Alert on symptoms. Causes help with diagnosis once someone is investigating. Paging on causes creates two problems.

• Many causes never become symptoms. A replica down with healthy failover is not an emergency.
• Many symptoms have causes you did not monitor. A symptom alert catches those too.

Rob Ewaschuk wrote My Philosophy on Alerting while an SRE at Google. He makes this case in detail. Read it in full.

An exception exists for causes that will certainly become symptoms soon. You have time to prevent the impact. A certificate expires in seven days. A disk will be full tomorrow. These deserve an alert. Usually, use a working-hours notification, not a page.

Frameworks for choosing signals

The four golden signals

The Google SRE book Monitoring Distributed Systems defines four signals for user-facing systems.

• Latency: how long requests take. Track failed requests separately, because fast errors make latency look good.
• Traffic: how much demand there is, such as requests per second.
• Errors: the rate of failed requests.
• Saturation: how full the service is, for the resource that runs out first.

RED

Tom Wilkie describes RED for request-driven services. It uses Rate, Errors and Duration. This is the golden signals without saturation, applied service by service.

USE

Brendan Gregg describes USE for resources like CPU, memory, disk and network. It uses Utilisation, Saturation and Errors. This method finds bottlenecks. It belongs more on dashboards than in paging rules.

Checks from outside

Internal metrics can look healthy while users cannot reach you. DNS may break. A certificate may expire. The load balancer may misroute. Add at least one check from outside your infrastructure.

• An uptime check that requests a real page or health endpoint, from more than one location.
• A synthetic check that walks a key journey, such as sign in or checkout.
• A heartbeat for scheduled work, which alerts when a job does not report in. See heartbeat monitoring.

What makes an alert worth a page

Before an alert pages someone, it must pass five checks.

1. It is urgent. Something bad is happening now or will within hours.
2. It is actionable. The person paged can do something about it.
3. It needs a human. If a script could fix it, write the script.
4. It is real. It rarely fires when nothing is wrong.
5. It says what is wrong. The title names the service and the symptom, and the body links to a dashboard and a runbook.

If an alert fails any of these, downgrade it to a working-hours notification or a ticket, or delete it.

Write the runbook with the alert

A runbook is the page an alert links to. It does not need to be long:

• What the alert means, in one sentence.
• How to check whether it is real.
• The first things to try.
• Who to escalate to if those do not work.

An alert without a runbook assumes the person paged already knows the system. At 3am that is often not true.

Tools

Use these categories to find the right tools.

• Metrics, APM and observability for internal signals.
• Uptime and synthetic monitoring for checks from outside.
• Error tracking for exceptions and crashes.