Alert notifications
This documentation topic discusses legacy alerting in Grafana. This will not be supported in future versions of Amazon Managed Grafana. You can migrate to Grafana alerting to use the latest alerting features. For more information, see one of the following topics.
For Grafana workspaces that support Grafana version 10.x, see Alerts in Grafana version 10.
For Grafana workspaces that support Grafana version 9.x, see Alerts in Grafana version 9.
For Grafana workspaces that support Grafana version 8.x, see Grafana alerting.
When an alert changes state, it sends out notifications. Each alert rule can have
multiple notifications. To add a notification to an alert rule, you first must add and
configure a notification channel.
This is done from the Notification channels page.
Adding a notification channel
-
In the side bar, pause on the Alerting (bell) icon, and then choose Notification channels.
-
Choose Add channel.
-
Fill out the fields or select options described in the following sections.
New notification channel fields
Default (send on all alerts)
-
Name – Enter a name for this channel. It will be displayed when users add notifications to alert rules.
-
Type – Select the channel type. For more information, see List of supported notifiers.
-
Default (send on all alerts) – When selected, this option sends a notification on this channel for all alert rules.
-
Disable Resolve Message – When selected, this option disables the resolve message [OK] that is sent when the alerting state returns to false.
-
Send reminders – When this option is selected, additional notifications (reminders) will be sent for alerts. You can specify how often reminders should be sent by using the number of seconds (s), minutes (m), or hours (h); for example,
30s,3m,5mor1h.
Important
Alert reminders are sent after rules are evaluated. Therefore, a reminder can't be sent more frequently than a configured alert rule evaluation interval.
The following examples show how often and when reminders are sent for a triggered alert.
| Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification) |
|---|---|---|
30s
|
15s
|
~30 seconds |
1m
|
5m
|
~5 minutes |
5m
|
15m
|
~15 minutes |
6m
|
20m
|
~24 minutes |
1h
|
15m
|
~1 hour |
1h
|
2h
|
~2 hours |
List of supported notifiers
| Name | Type | Supports images | Supports alert rule tags |
|---|---|---|---|
| Amazon Simple Notification Service | sns | No | Yes |
| OpsGenie |
opsgenie
|
No | Yes |
| PagerDuty |
pagerduty
|
No | Yes |
| Slack |
slack
|
No | No |
| VictorOps |
victorops
|
No | No |
Amazon Simple Notification Service
If you have enabled service-managed permissions and included Amazon SNS as a
notification channel for your workspace, you only need to provide the SNS Topic
ARN when you create your notification channel. In the Name
field, provide the name of the SNS topic that you have created. If you created
the workspace using service-managed permissions, the SNS topic name must be
prefixed with grafana for notifications to successfully publish to
the topic. If you selected customer-managed permissions when you created the
workspace, the SNS Topic name does not need to be prefixed with
grafana.
In the Topic field, copy and paste the ARN of the SNS topic. In the Message body format, you can choose either the JSON or the text option.
In the Optional AWS SNS Settings field, check the checkbox Include all tags in the message to see all the Grafana tags in the message body.
If you use customer-managed permissions for the workplace, the IAM role that you supply should include SNS Publish permissions for your SNS Topic.
Slack
To set up Slack, you must configure an incoming Slack webhook URL. For more
information, see Sending
messages using Incoming Webhooks
To include screenshots of the firing alerts in the Slack messages, you must
configure either the external image destination in Grafana or a bot integration
via Slack Apps. For more information about setting up a Slack bot integration,
see Follow Slack’s guide to set up
a bot integration
| Setting | Description |
|---|---|
| Url | Slack incoming webhook URL, or eventually the chat.postMessage |
| Username | Set the user name for the bot’s message. |
| Recipient | Use this to override the Slack recipient. You must provide
either a channel Slack ID, a user Slack ID, a user name
reference (@<user>, all lowercase, no whitespace), or a
channel reference (#<channel>, all lowercase, no white
space). If you use the chat.postMessage Slack API
endpoint, this is required. |
| Icon emoji | Provide an emoji to use as the icon for the bot’s message. For example, :smile: |
| Icon URL | Provide a URL to an image to use as the icon for the bot’s message. |
| Mention Users | Optionally mention one or more users in the Slack notification sent by Grafana. To see users, comma-separated, via their corresponding Slack IDs, choose the overflow button on each user’s Slack profile. |
| Mention Groups | Optionally mention one or more groups in the Slack notification sent by Grafana. You can see groups, comma-separated, via their corresponding Slack IDs (which you can get from each group’s Slack profile URL). |
| Mention Channel | Optionally mention either all channel members or only active ones. |
| Token | If provided, Amazon Managed Grafana will upload the generated image via
the Slack file.upload API operation, not the external image
destination. If you use the chat.postMessage Slack
API endpoint, this is required. |
If you are using the token for a slack bot, you have to invite the bot to the channel that you want to send notifications. Then add the channel to the recipient field.
PagerDuty
To set up PagerDuty, provide an integration key.
| Setting | Description |
|---|---|
| Integration Key | Integration key for PagerDuty. |
| Severity | Level for dynamic notifications; default is
critical (1) . |
| Auto resolve incidents | Resolve incidents in PagerDuty after the alert goes back to ok. |
| Message in details | Removes the Alert message from the PD summary field and puts it into custom details instead (2). |
Note
The tags Severity, Class, Group,
dedup_key, and Component have special meaning
in the PagerDuty
Common Event Format – PD-CEF
Note
Using Message In Details will change the structure of the
custom_details field in the PagerDuty Event. This might
break custom event rules in your PagerDuty rules if you rely on the fields
in payload.custom_details. Move any existing rules that use
custom_details.myMetric to
custom_details.queries.myMetric.
Note
Using dedup_key tag will override the Grafana generated
dedup_key with a custom key.
Configuring the link back to Grafana from alert notifications
All alert notifications contain a link back to the triggered alert in the Grafana workspace.
