Writing alert messages people can act on

The person who gets the alert may be half awake, on a phone, and may not know the service. The message has to tell them what broke, where, how bad, and what to do first, without a laptop. Most of that is decided when the alert rule is written, not when it fires. This guide covers what the alert says. It is a companion to what to alert on, which covers whether an alert should exist at all.

The title: what, where, how bad

Lock screens, SMS previews, chat notifications, and incident lists often display the title in isolation. It must communicate three facts in this specific order: the location of the issue, the symptom, and the severity.

Follow this pattern: <service> <env>: <symptom> <value> (threshold <threshold>). An example is checkout-api prod: 5xx rate 7.2% (threshold 2%). Name the symptom users feel, not the internal cause. See what to alert on.

Place important words at the start. Previews and short channels cut off the end. Avoid using only the rule's internal name like HighErrorRate. Generic phrases like "Alert triggered" or "Check failed" do not help. Status words without a subject confuse readers. Skip emoji in titles, as they may appear in SMS.

Keep the title shape identical for every fire of the same rule. Grouping and search work better. People learn to read it at a glance.

Fields to include in the body

Field	Example	Why
Service	`checkout-api`	Who owns it, which runbook
Environment	`prod`, `eu-west-1`	Staging alerts should not look like production
Value and threshold	`7.2%`, threshold `2%`	How far over, without opening a dashboard
Since when	`09:20 UTC`, for 10 minutes	New, or ongoing for hours
User impact	"Checkouts are failing for some users"	Decides the severity
Runbook link	`https://runbooks.example.com/...`	The first steps. See runbooks
Dashboard link	Filtered to this service and environment	One tap to the graph
Owner	`team: payments`	Who to escalate to
Severity	`critical`	Which response applies. See severity levels

Write times in UTC, or always state the time zone. Links must work from a phone and without a VPN where possible.

Length limits that matter

Where	Limit	Source
SMS, GSM-7	160 characters in one segment, 153 per segment when split	Twilio
SMS, UCS-2 (any emoji or non-GSM character)	70 in one segment, 67 per segment when split	Twilio
PagerDuty Events API v2 `summary`	1024 characters	PagerDuty
Slack message `text`	Keep under 4,000 characters. Longer may be split; above 40,000 it is truncated	Slack chat.postMessage
Slack attachments	100 per message	Slack chat.postMessage
Slack blocks	50 per message	Slack blocks
Slack header block	150 characters	Slack header block
Slack section block	3000 characters of text; up to 10 fields of 2000 each	Slack section block

One emoji forces the entire SMS into UCS-2 encoding. The character limit drops from 160 to 70. Some GSM-7 characters, such as { and ], count as two characters (Twilio GSM-7).

Slack uses the text field as the fallback shown in notifications when a message uses blocks. Always set it.

Tools cut silently. Alertmanager truncates the PagerDuty summary to 1024 characters and the Slack title to 1024 characters, and logs a warning (pagerduty.go, slack.go). Design the title to work at SMS length.

Before and after

The examples below are illustrations, not real alerts.

A poor message looks like this:

[FIRING:1] HighErrorRate
CRITICAL: check failed

This text fails to tell the reader which service is affected, which environment it is in, how bad the situation is, or what to do next.

A better message appears as a chat or push notification:

[FIRING] checkout-api prod: 5xx rate 7.2% (threshold 2%)
More than 2% of requests to checkout-api in prod have returned 5xx for at least 10 minutes. Checkouts are failing for some users.
Since: 09:20 UTC
Owner: payments
Runbook: https://runbooks.example.com/checkout-api/high-error-rate
Dashboard: https://grafana.example.com/d/checkout?var-env=prod

The same alert as an SMS (136 characters, GSM-7, one segment):

SEV1 checkout-api prod: 5xx rate 7.2% (threshold 2%) since 09:20 UTC. Runbook: https://runbooks.example.com/checkout-api/high-error-rate

Put the message in the alert rule

Write the message where the alert is defined, so every channel receives the same text. In Prometheus, annotations hold this content. Annotation values are templates. The $labels variable holds the alert's labels. The $value variable holds the evaluated value. See Prometheus alerting rules.

Use template functions to convert raw numbers into readable formats. humanizePercentage converts ratios to percentages. humanizeDuration formats durations. See the template reference.

The common annotation names are summary, description, and runbook_url. Grafana uses the same three names. See Grafana annotations. The dashboard_url below is a custom name. Any name works.

groups:
  - name: checkout-api
    rules:
      - alert: CheckoutApiHighErrorRate
        expr: |
          sum by (service, env) (rate(http_requests_total{service="checkout-api", code=~"5.."}[5m]))
            /
          sum by (service, env) (rate(http_requests_total{service="checkout-api"}[5m]))
            > 0.02
        for: 10m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: '{{ $labels.service }} {{ $labels.env }}: 5xx rate {{ $value | humanizePercentage }} (threshold 2%)'
          description: 'More than 2% of requests to {{ $labels.service }} in {{ $labels.env }} have returned 5xx for at least 10 minutes. Checkouts are failing for some users.'
          runbook_url: 'https://runbooks.example.com/checkout-api/high-error-rate'
          dashboard_url: 'https://grafana.example.com/d/checkout?var-env={{ $labels.env }}'

The rule passes promtool check rules (Prometheus v3.5.0). A promtool test rules unit test with a 7.2% error rate renders the summary as checkout-api prod: 5xx rate 7.2% (threshold 2%). The sum by (service, env) keeps those two labels on the result, so the templates can use them.

Format it in the notification template

Alertmanager groups alerts. Templates access .Status, .CommonAnnotations, .CommonLabels and .Alerts. Each alert exposes .Labels, .Annotations and .StartsAt (notification template reference). .CommonAnnotations contains only annotations shared by every alert in the group. If the group spans two environments, .CommonAnnotations.summary is empty. Group by the labels the summary uses, or range over .Alerts.

route:
  receiver: payments-slack
  group_by: [alertname, service, env]
receivers:
  - name: payments-slack
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: '#payments-alerts'
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        title_link: '{{ .CommonAnnotations.dashboard_url }}'
        text: |-
          {{ range .Alerts }}{{ .Annotations.description }}
          Since: {{ .StartsAt.Format "15:04 MST" }}
          Owner: {{ .Labels.team }}
          Runbook: {{ .Annotations.runbook_url }}
          {{ end }}

Set send_resolved to true. The Slack receiver defaults to false (configuration). The channel then shows when the problem ends.

Match group_by to the labels the summary uses. See Alertmanager routing and alert deduplication and grouping.

Checked with amtool check-config and amtool template render (Alertmanager v0.35.0).

Resolved messages

State what recovered and how long it lasted. Use the same title shape as the firing message. This pairs the resolve with the alert in a channel or incident timeline.

Send the resolve to every channel the alert used. Otherwise someone may keep working on a fixed problem. In the PagerDuty format the resolve needs the same dedup_key as the trigger. See the PagerDuty Events API v2 format.

Checklist

The title follows the service, environment, symptom, value, and threshold sequence.
The title stays under 160 GSM-7 characters for SMS compatibility and excludes emoji.
The body specifies the start time, user impact, owner, runbook link, and dashboard link.
All times carry a time zone.
Links function correctly on mobile devices.
Annotations derive from label-based templates rather than manual typing.
Grouping aligns with the labels in the summary.
Resolved notifications remain enabled.
Review the last ten alerts as an on-call engineer would on a phone. Rewrite any message that lacks actionable information.

See choosing alert notification channels for where alerts go next.