Monitoring an integration

Integration states

The following integration states describe the integration:

Viewing Amazon CloudWatch logs for an integration

AWS Glue zero-ETL integrations generate Amazon CloudWatch logs for visibility into your data movement. Log events relating to each successful ingestion or any failures experienced due to problematic data records at source, or data write errors due to schema changes or insufficient permissions are emitted to a default log group created in a customer account.

Log Group Structure

The logging architecture implements a hierarchical structure of integration instances. Each integration instance maintains dedicated log streams within a centralized log group specifically designed for zero-ETL operations.

Log Stream Identification and Path

Each integration instance is assigned a unique log stream based on its Integration ID, extracted from the Integration ARN. The complete log stream path follows this structure:

Format: /aws-glue/zeroETL-integrations/logs/{IntegrationId}

Example: Given an Integration ARN:

arn:aws:glue:us-east-1:123456789012:integration:03cabe77-79e7-4b7a-b3da-8c160bea6bbf

The corresponding full log stream path would be:

/aws-glue/zeroETL-integrations/logs/03cabe77-79e7-4b7a-b3da-8c160bea6bbf

This architectural approach ensures segregation of log events per integration while maintaining a unified logging framework for the entire zero-ETL infrastructure. The granular logging strategy facilitates efficient troubleshooting, audit compliance, and operational monitoring of data movement processes.

IAM permissions required to enable logging

When creating your integration, the following IAM permissions are needed by the source and target roles to enable CloudWatch logging for an integration. AWS Glue zero-ETL integrations use these permissions provided in the source and target roles to emit CloudWatch logs to customer accounts.

{
"Version": "2012-10-17",
    "Statement": [
        {
"Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
             ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Log messages

Log format: zero-ETL integrations emit four types of log messages:

// Ingestion started
{
"integrationArn": "arn:aws:glue:us-east-2:123456789012:integration/1a012bba-123a-1bba-ab1c-173de3b12345",
...
    "messageType": "IngestionStarted",
    "details": {
        "tableName": "testDDBTable",
        "message": "Ingestion Job started"
    }
}
// Data processing stats on successful table ingestion
{
...
    "messageType": "IngestionProcessingStats",
    "details": {
        "tableName": "testDDBTable",
        "insert_count": 100,
        "update_count": 10,
        "delete_count": 10   
    }
}
// Ingestion failure logs for failed table-processing
{
...
    "messageType": "IngestionFailed",
    "details": {
        "tableName": "testDDBTable",
        "errorMessage": "Failed to ingest data with error: Target Glue database not found.",
        "error_code" : "client_error"
    }
}
// Ingestion completed notification with lastSyncedTimestamp
{
...
    "messageType": "IngestionCompleted",
    "details": {
        "tableName": "testDDBTable",
        "message": "Ingestion Job completed"
        "lastSyncedTimestamp": "1132344255745"
    }
}

Viewing Amazon CloudWatch metrics for an integration

AWS Glue zero-ETL provides real-time operational insights through CloudWatch metrics, enabling proactive monitoring of data integration processes without direct querying of target Iceberg tables. When enabled by adding appropriate permissions on source and target processing role, CloudWatch metrics are automatically emitted to the AWS/Glue/ZeroETL namespace after completion of each table ingestion operation. You can setup alarms on your CloudWatch metrics to get notified when a particular Ingestion Job fails.

Metric Specifications

Ingestion Processing Metrics:

Operational Status Metrics:

Metric Dimensions:

IAM permissions required to enable CloudWatch Metrics

When creating your integration, the following IAM permissions are needed by the source and target roles to enable CloudWatch Metrics for an integration. AWS Glue zero-ETL integrations use these permissions provided in the source and target roles to emit CloudWatch Metrics to customer accounts.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "cloudwatch:namespace": "AWS/Glue/ZeroETL"
                }
            }
        }
    ]
}
		

Managing event notifications with Amazon EventBridge

Zero-ETL integrations use Amazon EventBridge to manage event notifications to keep you up-to-date regarding changes in your integrations. Amazon EventBridge is a serverless event bus service that you can use to connect your applications with data from a variety of sources. In this case, the event source is AWS Glue. Events, which are monitored changes in an environment, are sent to EventBridge from AWS Glue automatically. Events are delivered in near real time.

EventBridge provides an environment for you to write event rules, which can specify actions to take for specific events. You can also set up targets, which are resources that EventBridge can send an event to. A target can include an API destination, an Amazon CloudWatch log group, and others. For more information about rules, see Amazon EventBridge rules. For more information about targets, see Amazon EventBridge targets.

To capture all Zero-ETL notifications, create an Eventbridge rule which matches the following:

{
  "source": [{
    "prefix": "aws.glue-zero-etl“
  }],
  "detail-type": [{
    "prefix": "Glue Zero ETL“
  }]
}

The following table includes zero-ETL integration events, with additional metadata: