Alertmanager routing, grouping and inhibition explained

Prometheus evaluates alert rules. It does not decide who hears about them. That is the job of Alertmanager. It deduplicates, groups and routes alerts to receivers such as email, PagerDuty or Opsgenie. It also handles silences and inhibition. All of this lives in one file, alertmanager.yml. This guide walks through the parts that decide who gets notified and how often. Every default below comes from the Alertmanager configuration reference. The complete example near the end passes amtool check-config on Alertmanager v0.28.1.

How an alert moves through Alertmanager

An alert is identified by its labels. Alerts with the same label set are the same alert and are deduplicated. Annotations are not part of the identity; the latest received values win (clients).

• The routing tree picks the receiver, or several receivers.
• Each route groups alerts by the labels in group_by and times the notifications.
• Inhibition rules and silences mute alerts before anything is sent.
• The receiver delivers the notification to Slack, PagerDuty, a webhook or another integration.

The routing tree

Every alert enters at the top-level route. The root must match all alerts, so it has no matchers, and it must name a receiver.

The alert is then checked against the child routes in order. With continue: false (the default) it stops at the first child that matches. With continue: true on a matching child, it keeps checking the siblings after it, so one alert can reach several receivers. If no child matches, the alert is handled by the current node. Children inherit settings they do not set, such as receiver, group_by and the timers.

A route matches with matchers, a list in which every matcher must match. The operators are = (equals), != (not equal), =~ (regex match) and !~ (regex non-match). Regexes are anchored at both ends, so service=~"mysql|cassandra" matches only those two values. The older match and match_re fields are deprecated; use matchers.

matchers:
  - team="payments"
  - severity=~"critical|warning"

continue and the catch-all trap

A common pattern places a first child route with no matchers and continue: true. This copies every alert to an audit or archive receiver. That child matches all alerts. Because a child matched, the alert never falls back to the root receiver. Testing the example config without its last route showed a warning from team search reaching only audit-webhook and never platform-slack. The fix is an explicit catch-all child at the end of the list, with no matchers, pointing at the default receiver. The example below has one.

Grouping

Grouping consolidates multiple alerts into a single notification. The documented case is a network partition where hundreds of instances lose database connectivity simultaneously. Grouping by cluster and alertname sends one compact notification that still lists every affected instance (concepts).

• group_by lists the labels that define a group. Alerts with the same values for those labels on the same route share a group and a notification.
• Group by the labels a responder acts on together, such as alertname, cluster and service. Leave out per-instance labels such as instance or pod, or every instance gets its own notification.
• The special value group_by: ['...'] groups by all labels, which turns grouping off. The reference says this is unlikely to be what you want unless volume is very low or a downstream tool does its own grouping.

The deduplication and grouping guide covers how the group becomes the dedup key that PagerDuty or Opsgenie sees.

The three timers

Setting	Default	What it controls	Typical value in the reference
group_wait	30s	How long a new group waits before its first notification, to collect more alerts or let an inhibiting alert arrive	0s to a few minutes
group_interval	5m	How long to wait before notifying about new alerts added to a group that has already notified	5m or more
repeat_interval	4h	How long to wait before sending the same notification again	3h or more

• repeat_interval should be a multiple of group_interval, because it is checked after each group_interval.
• repeat_interval is also bounded by the --data.retention flag (default 120h). Notifications are resent after whichever comes first.

Illustration with the defaults:

• 00:00 alert A fires and a new group starts.
• 00:30 the first notification goes out, listing A.
• 02:00 alert B joins the same group.
• 05:30 the next group_interval tick sends a notification listing A and B.
• While nothing changes, no more notifications go out until repeat_interval has passed.

Use a shorter group_wait like 10s for pages. Use longer timers for chat channels.

Inhibition

An inhibit rule mutes target alerts while a source alert is firing.

• source_matchers select the alerts that do the muting.
• target_matchers select the alerts that get muted.
• equal lists labels that must have the same value on source and target. A missing label and an empty label count as equal, so if none of the equal labels exist on either side, the rule applies to everything it matches.
• An alert that matches both sides cannot be inhibited by another alert that also matches both sides, including itself. The reference recommends matchers that never match both sides.
• The older source_match and target_match fields are deprecated.

The two rules in the example do the common jobs. A critical alert mutes the warning for the same alertname, cluster and service. A ClusterUnreachable alert mutes every other alert from that cluster. In a test on v0.28.1, the muted alerts showed as suppressed with the source alert in inhibitedBy.

Receivers

A receiver pairs a name with one or more integrations. Supported types include email_configs, slack_configs, pagerduty_configs, opsgenie_configs, webhook_configs, msteams_configs, and telegram_configs. Routes refer to receivers by name. Every name a route uses must exist in the configuration.

• send_resolved controls whether a resolved notification is sent. It defaults to true for PagerDuty, Opsgenie, and webhooks. It defaults to false for Slack and email. Set it explicitly to avoid surprises.
• Keep secrets out of the file. Use the _file variants, such as routing_key_file for PagerDuty and api_url_file for Slack.

How to word what the receiver sends is covered in alert message design. Channel choice is in notification channels.

Silences

You use silences to mute alerts based on matchers for a set duration. You can create them in the web UI, via amtool, or through the API. Do not store silences in alertmanager.yml. The guide silences and maintenance windows covers the commands and associated risks.

A complete alertmanager.yml

This configuration handles routing for platform and payments teams. Critical alerts trigger pages. All other alerts route to chat channels. Every alert also copies to an audit webhook. The file passes amtool check-config on Alertmanager v0.28.1.

global:
  resolve_timeout: 5m

route:
  receiver: platform-slack
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Copy every alert to the audit webhook, then keep matching.
    - receiver: audit-webhook
      continue: true

    # Critical payments alerts page the payments rotation.
    - matchers:
        - team="payments"
        - severity="critical"
      receiver: payments-pager
      group_wait: 10s
      repeat_interval: 1h

    # Everything else from payments goes to their channel.
    - matchers:
        - team="payments"
      receiver: payments-slack

    # Critical alerts from any other team page the platform rotation.
    - matchers:
        - severity="critical"
      receiver: platform-pager
      group_wait: 10s
      repeat_interval: 1h

    # Anything not matched above goes to the platform channel.
    - receiver: platform-slack

inhibit_rules:
  # A critical alert mutes the warning for the same problem.
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'cluster', 'service']

  # An unreachable cluster mutes every other alert from that cluster.
  - source_matchers:
      - alertname="ClusterUnreachable"
    target_matchers:
      - alertname!="ClusterUnreachable"
    equal: ['cluster']

receivers:
  - name: platform-slack
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook-url
        channel: '#platform-alerts'
        send_resolved: true

  - name: platform-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/platform-pd-key

  - name: payments-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/payments-pd-key

  - name: payments-slack
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook-url
        channel: '#payments-alerts'
        send_resolved: true

  - name: audit-webhook
    webhook_configs:
      - url: http://alert-audit.internal:8080/alertmanager

Test the config before you ship it

amtool ships with every Alertmanager release (amtool README). Check the syntax, then check where sample alerts go:

amtool check-config alertmanager.yml
amtool config routes --config.file=alertmanager.yml
amtool config routes test --config.file=alertmanager.yml team=payments severity=critical
amtool config routes test --config.file=alertmanager.yml team=search severity=warning

Against the example, the two tests print audit-webhook,payments-pager and audit-webhook,platform-slack. Add --verify.receivers=audit-webhook,payments-pager to make the command exit with code 1 when the result differs, which turns a route into a CI test.

Alertmanager reloads its config on SIGHUP or an HTTP POST to /-/reload. A config that is not well-formed is not applied, and the error is logged. To test routing end to end, see testing escalation policies. Tools that route and group alerts are listed in the alert routing directory.