Create and manage alerting resources using Terraform

Create a new service account for your CI pipeline.

Assign the role “Access the alert rules Provisioning API.”

Create a new service account token.

Name and save the token for use in Terraform.

Copy this code block into a .tf file on your local machine. Replace <slack-webhook-url> with your Slack webhook URL (or other contact point details).

This example creates a contact point that sends alert notifications to Slack.

resource "grafana_contact_point" "my_slack_contact_point" {
    name = "Send to My Slack Channel"

    slack {
        url = <slack-webhook-url>
        text = <<EOT
{{ len .Alerts.Firing }} alerts are firing!

Alert summaries:
{{ range .Alerts.Firing }}
{{ template "Alert Instance Template" . }}
{{ end }}
EOT
    }
}

Enter text for your notification in the text field.

The text field supports Go-style templating. This enables you to manage your Grafana Alerting notification templates directly in Terraform.

Run the command terraform apply.

Go to the Grafana UI and check the details of your contact point.

You cannot edit resources provisioned via Terraform from the UI. This ensures that your alerting stack always stays in sync with your code.

Click Test to verify that the contact point works correctly.

Copy this code block into a .tf file on your local machine.

In this example, the alerts are grouped by alertname, which means that any notifications coming from alerts which share the same name, are grouped into the same Slack message.

If you want to route specific notifications differently, you can add sub-policies. Sub-policies allow you to apply routing to different alerts based on label matching. In this example, we apply a mute timing to all alerts with the label a=b.

resource "grafana_notification_policy" "my_policy" {
    group_by = ["alertname"]
    contact_point = grafana_contact_point.my_slack_contact_point.name

    group_wait = "45s"
    group_interval = "6m"
    repeat_interval = "3h"

    policy {
        matcher {
            label = "a"
            match = "="
            value = "b"
        }
        group_by = ["..."]
        contact_point = grafana_contact_point.a_different_contact_point.name
        mute_timings = [grafana_mute_timing.my_mute_timing.name]

        policy {
            matcher {
                label = "sublabel"
                match = "="
                value = "subvalue"
            }
            contact_point = grafana_contact_point.a_third_contact_point.name
            group_by = ["..."]
        }
    }
}

In the mute_timings field, link a mute timing to your notification policy.

Run the command terraform apply.

Go to the Grafana UI and check the details of your notification policy.

Click Test to verify that the notification point is working correctly.

Copy this code block into a .tf file on your local machine.

In this example, alert notifications are muted on weekends.

resource "grafana_mute_timing" "my_mute_timing" {
    name = "My Mute Timing"

    intervals {
        times {
          start = "04:56"
          end = "14:17"
        }
        weekdays = ["saturday", "sunday", "tuesday:thursday"]
        months = ["january:march", "12"]
        years = ["2025:2027"]
    }
}

Run the command terraform apply.

Go to the Grafana UI and check the details of your mute timing.

Reference your newly created mute timing in a notification policy using the mute_timings field. This will apply your mute timing to some or all of your notifications.

Click Test to verify that the mute timing is working correctly.

Create a data source to query and a folder to store your rules in.

In this example, the Configure a TestData data source for testing data source is used.

Alerts can be defined against any backend datasource in Grafana.

resource "grafana_data_source" "testdata_datasource" {
    name = "TestData"
    type = "testdata"
}

resource "grafana_folder" "rule_folder" {
    title = "My Rule Folder"
}

Define an alert rule.

For more information on alert rules, refer to how to create Grafana-managed alerts.

Create a rule group containing one or more rules.

In this example, the grafana_rule_group resource group is used.

resource "grafana_rule_group" "my_rule_group" {
    name = "My Alert Rules"
    folder_uid = grafana_folder.rule_folder.uid
    interval_seconds = 60
    org_id = 1

    rule {
        name = "My Random Walk Alert"
        condition = "C"
        for = "0s"

        // Query the datasource.
        data {
            ref_id = "A"
            relative_time_range {
                from = 600
                to = 0
            }
            datasource_uid = grafana_data_source.testdata_datasource.uid
            // `model` is a JSON blob that sends datasource-specific data.
            // It's different for every datasource. The alert's query is defined here.
            model = jsonencode({
                intervalMs = 1000
                maxDataPoints = 43200
                refId = "A"
            })
        }

        // The query was configured to obtain data from the last 60 seconds. Let's alert on the average value of that series using a Reduce stage.
        data {
            datasource_uid = "__expr__"
            // You can also create a rule in the UI, then GET that rule to obtain the JSON.
            // This can be helpful when using more complex reduce expressions.
            model = <<EOT
{"conditions":[{"evaluator":{"params":[0,0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["A"]},"reducer":{"params":[],"type":"last"},"type":"avg"}],"datasource":{"name":"Expression","type":"__expr__","uid":"__expr__"},"expression":"A","hide":false,"intervalMs":1000,"maxDataPoints":43200,"reducer":"last","refId":"B","type":"reduce"}
EOT
            ref_id = "B"
            relative_time_range {
                from = 0
                to = 0
            }
        }

        // Now, let's use a math expression as our threshold.
        // We want to alert when the value of stage "B" above exceeds 70.
        data {
            datasource_uid = "__expr__"
            ref_id = "C"
            relative_time_range {
                from = 0
                to = 0
            }
            model = jsonencode({
                expression = "$B > 70"
                type = "math"
                refId = "C"
            })
        }
    }
}

Go to the Grafana UI and check your alert rule.

You can see whether the alert rule is firing. You can also see a visualization of each of the alert rule’s query stages.

When the alert fires, Grafana routes a notification through the policy you defined.

For example, if you chose Slack as a contact point, Grafana’s embedded Alertmanager automatically posts a message to Slack.