What is alert escalation?

Alert escalation is what happens between a failed check and a fix. A monitoring tool raises an alert. A system decides who must respond. That person receives a notification. If they do not acknowledge it within a set time, the next person in the chain is notified. This continues until someone takes action.

The term covers two related practices. Automatic escalation moves an unacknowledged alert up a chain of people after fixed waits. This is what an "escalation policy" usually means. Manual escalation occurs when a responder decides they need help. They pull in more people, a specialist team, or a manager. Incident processes define when to do this.

This guide covers the first practice. The second is covered in incident response basics.

The parts

Every escalation setup shares the same components. This holds true whether you use a shell script or a commercial on-call product.

An alert source. Something detects that a condition is bad. Examples include an uptime check, a Prometheus alert rule, a missed cron heartbeat, or a spike in errors. The source sends the alert to a destination, usually via a webhook or an email.

Routing. Rules determine which service or team owns the alert. Routing logic can use the source, labels such as team=payments, or text within the alert. Grouping and deduplication also happen here, so that fifty failing checks for one outage become one notification.

A schedule. This defines who is on call right now for a specific service. Schedules are built from rotations. A rotation is an ordered list of people, a shift length, and a handover time. See on-call rotation patterns for details.

An escalation policy. This is the ordered list of levels. It specifies who is notified first, how long to wait for an acknowledgement, and who is notified next. A level usually points at a schedule rather than a named person. This means the policy does not need editing when the rota changes.

Notification channels. These are the methods by which a person hears about the alert. Options include push notifications, SMS, phone calls, chat messages, and email. Channels differ in their likelihood of waking someone up.

Acknowledge and resolve. Acknowledging tells the system "this is mine" and stops the escalation. Resolving closes the alert or incident. Many tools also resolve alerts automatically when the source reports that the condition has cleared.

A worked example

A payments API starts returning errors at 02:10.

1. The metrics system evaluates an alert rule. The error rate has been above the threshold for five minutes, so at 02:15 it fires and sends a webhook.
2. The on-call tool matches the alert to the payments service and opens an incident.
3. Level 1 of the payments escalation policy targets the primary on-call schedule. Tonight that is Priya. She gets a push notification and, if she has chosen it, a phone call.
4. The level waits 10 minutes for an acknowledgement. Priya's phone is on silent in another room, so nothing happens.
5. At 02:25 the policy moves to level 2, the secondary schedule. Sam gets the page and acknowledges at 02:27. Escalation stops.
6. Sam fixes the problem and resolves the incident. The next morning the team reviews why the primary page was missed.

The names and times are an illustration, not a real incident. The mechanics are the same in almost every tool.

Why it matters

Without escalation, an alert reaching one person relies on that person. People sleep through phones, lose signal, get ill and go on holiday. Escalation turns "someone should see this" into "someone will see this within a known time".

It also lets everyone else switch off. When the rules say who is responsible and who is next, the rest of the team does not need to watch a shared channel all night.

Where to go next

• How to design an escalation policy: levels, waits and repeats.
• On-call rotation patterns: who is on call and when.
• On-call and alert escalation tools: the tools that run schedules and policies.