Alert fatigue: causes and fixes

Alert fatigue hits when volume outpaces action. You mute channels, snooze pages, and treat the next alert as noise. A real incident then goes missed. Tools alone do not fix this. You must decide every page is worth it, then remove the rest.

Common causes

• Cause-based alerts. Paging on CPU, memory, or queue depth when users are fine. See what to alert on.
• Thresholds nobody tuned. Defaults copied from a template, set too tight for this system.
• Flapping. A check crosses its threshold back and forth. It sends open, resolve, open, resolve.
• Duplicates. One outage fires alerts from the uptime checker, the metrics system, the logs, and the error tracker. Each pages separately.
• Everything is urgent. Warnings and critical alerts go through the same channel with the same sound.
• Nobody owns cleanup. Alerts are added after every incident and never removed.

Measure it first

Before changing anything, count, for the last four weeks:

• How many pages were there, and how many were at night?
• How many led to someone doing something?
• Which alerts fired most often?

A short list of the top ten noisiest alerts usually shows where most of the noise comes from.

Fixes

Delete

Delete alerts that fire often but trigger no action. This is the most effective fix. Move the alert to a dashboard panel instead. If you worry about losing visibility, downgrade it to a ticket first. Watch to see if anyone misses it.

Wait before firing

Most alert rules can require a condition to hold for a period before firing. Prometheus rules use for: 5m, for example. This removes blips and most flapping. It costs a few minutes of detection time.

Group and deduplicate

Combine alerts with the same cause into one notification. Prometheus Alertmanager groups by labels such as cluster and service. Most on-call tools deduplicate by an alert key. One outage should produce one page.

Inhibit dependent alerts

If the database is down, every service that uses it will alert too. Inhibition rules suppress the downstream alerts while the upstream one is firing. Alertmanager supports this directly. Some on-call tools have dependency settings for the same purpose.

Route by urgency

Send warnings to a chat channel or a ticket queue in working hours. Page only for critical alerts. See severity levels and the urgency section of escalation policy design.

Silence planned work

Put deployments and maintenance in silences or maintenance windows. This prevents expected failures from paging you.

Review every week

A short weekly review of last week's pages keeps noise from growing back. Run the review with whoever was on call. For each page, ask: was it real, was it actionable, and should it change?

A target to aim for

There is no universal number. The Google SRE book suggests on-call engineers handle at most two incidents per 12-hour shift. This leaves time to respond properly and follow up. If pages routinely exceed that, the alerts need work before the people do.

Tools

Use alert routing and notification delivery for grouping, deduplication and silences. Use on-call and escalation for urgency-based routing.