Continuous logging for AWS Glue jobs - AWS Glue

Continuous logging for AWS Glue jobs

AWS Glue provides real-time, continuous logging for AWS Glue jobs. You can view real-time Apache Spark job logs in Amazon CloudWatch, including driver logs, executor logs, and an Apache Spark job progress bar. Viewing real-time logs provides you with a better perspective on the running job.

When you start an AWS Glue job, it sends the real-time logging information to CloudWatch (every 5 seconds and before each executor termination) after the Spark application starts running. You can view the logs on the AWS Glue console or the CloudWatch console dashboard.

The continuous logging feature includes the following capabilities:

  • Continuous logging

  • A custom script logger to log application-specific messages

  • A console progress bar to track the running status of the current AWS Glue job

For information about how continuous logging is supported in AWS Glue version 2.0, see Running Spark ETL Jobs with Reduced Startup Times.

You can restrict access to CloudWatch log groups or streams for IAM roles to read the logs. For more details on restricting access, see Using identity-based policies (IAM policies) for CloudWatch Logs in the CloudWatch documentation.

Note

You may incur additional charges when you enable continuous logging and additional CloudWatch log events are created. For more information, see Amazon CloudWatch pricing .

Topics