Monitoring dashboards and alarms on Amazon MWAA

The environment details page for an Amazon Managed Workflows for Apache Airflow environment includes a built-in CloudWatch metrics dashboard and an Environment alarms table. You can use these features to monitor the health of your environment directly in the Amazon MWAA console, without any additional setup.

Apache Airflow exposes metrics for several processes, including the number of DAG processes, DAG bag size, currently running tasks, task failures, and successes. When you create an environment, Apache Airflow automatically sends metrics for your Amazon MWAA environment to CloudWatch. In addition to the built-in dashboard, you can build custom dashboards in Amazon CloudWatch and add alarms for particular metrics. When an alarm is on a dashboard, it turns red when it enters the ALARM state. This makes it easier to monitor the health of your Amazon MWAA environment proactively.

View the metrics dashboard

The environment details page includes a built-in CloudWatch metrics dashboard, shown in the console under CloudWatch metrics. This embedded CloudWatch dashboard displays charts for your environment without any additional setup.

To open the full set of metrics in the CloudWatch console, choose View all in CloudWatch.

The dashboard displays metrics from the following two CloudWatch namespaces:

You can use the following controls to change what the dashboard displays:

For the full list of metrics that Amazon MWAA publishes, see Apache Airflow environment metrics in CloudWatch.

Metrics shown on the dashboard

The dashboard organizes charts into six sections that you can filter with Metric sections. It shows a curated set of environment-wide metrics; charts come from the AmazonMWAA namespace (Apache Airflow metrics) and the AWS/MWAA namespace (container, queue, and database metrics). The following table lists the charts in each section and the CloudWatch metrics each one displays.

The dashboard shows aggregate, environment-wide metrics. It does not chart high-cardinality metrics that Apache Airflow publishes per DAG, task, pool, operator, or DAG file, and it charts container CPU and memory but not every container metric. For the complete list of published metrics, see Apache Airflow environment metrics in CloudWatch and Container, queue, and database metrics for Amazon MWAA.

Create and monitor alarms

The environment details page includes an Alarms pane where you create and manage your alarms in CloudWatch, and then monitor them in context. The pane shows an Environment alarms table that lists the CloudWatch alarms attributed to the environment. Amazon MWAA matches these alarms by CloudWatch namespace and the Environment dimension, so the table includes alarms that you author yourself, not only the recommended alarms.

The pane header shows an In alarm count of how many of the environment's alarms are currently in the ALARM state. The Environment alarms table has the following columns.

If the environment has no alarms, the Environment alarms table is empty and provides guidance to create the recommended alarms to start monitoring your environment.

The Alarms pane provides the following two buttons:

After CloudFormation creates the stack, the alarms appear in the Environment alarms table and in the CloudWatch console. For more information about what each alarm state means, see Alarm states overview.

Recommended alarms

When you choose Create recommended alarms, the CloudFormation template creates the following alarms. CloudWatch evaluates all of these alarms over 5-minute (300-second) periods.

Alarm states overview

A metric alarm has the following possible states:

Example custom dashboards and alarms

You can define your own dashboards and alarms as infrastructure as code to build custom dashboards programmatically or to standardize monitoring across multiple Amazon MWAA environments. As an alternative to the built-in dashboard on the environment details page, the following tutorial and CloudFormation template create a custom monitoring dashboard. This dashboard displays charts of selected metrics for your Amazon MWAA environment.

About these metrics

The following list describes each of the metrics created in the custom dashboard by the tutorial and template definitions in this section.

About the dashboard

The following image displays the monitoring dashboard created by the tutorial and template definition in this section.

Using AWS tutorials

You can use the following AWS tutorial to automatically create a health status dashboard for any Amazon MWAA environments that are currently deployed. It also creates CloudWatch alarms for unhealthy workers and scheduler heartbeat failures across all Amazon MWAA environments.

Using CloudFormation

You can use the CloudFormation template definition in this section to create a monitoring dashboard in CloudWatch, then add alarms on the CloudWatch console to receive notifications when a metric surpasses a particular threshold. To create the stack using this template definition, refer to Creating a stack on the CloudFormation console. To add an alarm to the dashboard, refer to Using alarms.

AWSTemplateFormatVersion: "2010-09-09"
Description: Creates MWAA Cloudwatch Dashboard
Parameters:
  DashboardName:
    Description: Enter the name of the CloudWatch Dashboard
    Type: String
  EnvironmentName:
    Description: Enter the name of the MWAA Environment
    Type: String    
Resources:
  BasicDashboard:
    Type: AWS::CloudWatch::Dashboard
    Properties:
      DashboardName: !Ref DashboardName
      DashboardBody:
        Fn::Sub: '{
              "widgets": [
                  {
                      "type": "metric",
                      "x": 0,
                      "y": 0,
                      "width": 12,
                      "height": 6,
                      "properties": {
                          "view": "timeSeries",
                          "stacked": true,
                          "metrics": [
                              [
                                  "AmazonMWAA",
                                  "QueuedTasks",
                                  "Function",
                                  "Executor",
                                  "Environment",
                                  "${EnvironmentName}"
                              ]
                          ],
                          "region": "${AWS::Region}",
                          "title": "QueuedTasks ${EnvironmentName}",
                          "period": 300
                      }
                  },
                  {
                      "type": "metric",
                      "x": 0,
                      "y": 6,
                      "width": 12,
                      "height": 6,
                      "properties": {
                          "view": "timeSeries",
                          "stacked": true,
                          "metrics": [
                              [
                                  "AmazonMWAA",
                                  "RunningTasks",
                                  "Function",
                                  "Executor",
                                  "Environment",
                                  "${EnvironmentName}"
                              ]
                          ],
                          "region": "${AWS::Region}",
                          "title": "RunningTasks ${EnvironmentName}",
                          "period": 300
                      }
                  },
                  {
                      "type": "metric",
                      "x": 12,
                      "y": 6,
                      "width": 12,
                      "height": 6,
                      "properties": {
                          "view": "timeSeries",
                          "stacked": true,
                          "metrics": [
                              [
                                  "AmazonMWAA",
                                  "SchedulerHeartbeat",
                                  "Function",
                                  "Scheduler",
                                  "Environment",
                                  "${EnvironmentName}"
                              ]
                          ],
                          "region": "${AWS::Region}",
                          "title": "SchedulerHeartbeat ${EnvironmentName}",
                          "period": 300
                      }
                  },
                  {
                      "type": "metric",
                      "x": 12,
                      "y": 0,
                      "width": 12,
                      "height": 6,
                      "properties": {
                          "view": "timeSeries",
                          "stacked": true,
                          "metrics": [
                              [
                                  "AmazonMWAA",
                                  "TasksPending",
                                  "Function",
                                  "Scheduler",
                                  "Environment",
                                  "${EnvironmentName}"
                              ]
                          ],
                          "region": "${AWS::Region}",
                          "title": "TasksPending ${EnvironmentName}",
                          "period": 300
                      }
                  },
                  {
                      "type": "metric",
                      "x": 0,
                      "y": 12,
                      "width": 24,
                      "height": 6,
                      "properties": {
                          "view": "timeSeries",
                          "stacked": true,
                          "region": "${AWS::Region}",
                          "metrics": [
                              [
                                  "AmazonMWAA",
                                  "TotalParseTime",
                                  "Function",
                                  "DAG Processing",
                                  "Environment",
                                  "${EnvironmentName}"
                              ]
                          ],
                          "title": "TotalParseTime  ${EnvironmentName}",
                          "period": 300
                      }
                  }
              ]
          }'

Deleting metrics and dashboards

If you delete an Amazon MWAA environment, the corresponding dashboard is also deleted. CloudWatch metrics are stored for fifteen (15) months and can not be deleted. The CloudWatch console limits the search of metrics to two (2) weeks after a metric is last ingested to ensure that the most up to date instances are shown for your Amazon MWAA environment. To learn more, refer to Amazon CloudWatch FAQs.

What's next?