Monitoring jobs

You can use Amazon CloudWatch Events to track jobs that run on an Amazon EMR on EKS virtual cluster. You can use events to track the activity and health of a jobs that you run on a virtual cluster. The topics that follow show you ways to configure monitoring effectively to maintain the health of your resources.

Monitor jobs with Amazon CloudWatch Events

Amazon EMR on EKS emits events when the state of a job run changes. Each event provides information, such as the date and time when the event occurred, along with further details about the event, such as the virtual cluster ID and the ID of the job run that was affected.

You can use events to track the activity and health of a jobs that you run on a virtual cluster. You can also use Amazon CloudWatch Events to define an action to take when a job run generates an event that matches a pattern that you specify. Events are useful for monitoring a specific occurrence during the lifecycle of a job run. For example, you can monitor when a job run changes state from submitted to running. For more information about CloudWatch Events, see the Amazon EventBridge User Guide.

The following table lists Amazon EMR on EKS events along with the state or state change that the event indicates, the severity of the event, and event messages. Each event is represented as a JSON object that is sent automatically to an event stream. The JSON object includes further details about the event. The JSON object is particularly important when you set up rules for event processing using CloudWatch Events because rules seek to match patterns in the JSON object. For more information, see Amazon EventBridge event patterns and Amazon EMR on EKS Events in the Amazon EventBridge User Guide.

Automate Amazon EMR on EKS with CloudWatch Events

You can use Amazon CloudWatch Events to automate your AWS services to respond to system events such as application availability issues or resource changes. Events from AWS services are delivered to CloudWatch Events in near real time. You can write simple rules to indicate which events are of interest to you and what automated actions to take when an event matches a rule. The actions that can be automatically triggered include the following:

Some examples of using CloudWatch Events with Amazon EMR on EKS include the following:

CloudWatch Events for "detail-type:" "EMR Job Run State Change" are generated by Amazon EMR on EKS for SUBMITTED, RUNNING, CANCELLED, FAILED and COMPLETED state changes.

Example: Set up a rule that invokes Lambda

Use the following steps to set up a CloudWatch Events rule that invokes Lambda when there is an "EMR Job Run State Change" event.

aws events put-rule \
--name cwe-test \
--event-pattern '{"detail-type": ["EMR Job Run State Change"]}'

Add the Lambda function that you own as a new target and give CloudWatch Events permission to invoke the Lambda function as follows. Replace 123456789012 with your account ID.

aws events put-targets \
--rule cwe-test \
--targets Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:MyFunction

aws lambda add-permission \
--function-name MyFunction \
--statement-id MyId \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com

Monitor job’s driver pod with a retry policy using Amazon CloudWatch Events

Using CloudWatch events, you can monitor driver pods that have been created in jobs that have retry policies. For more information, see Monitoring a job with a retry policy in this guide.