Using job retry policies

In Amazon EMR on EKS versions 6.9.0 and later, you can set a retry policy for your job runs. Retry policies cause a job driver pod to be restarted automatically if it fails or is deleted. This makes long-running Spark streaming jobs more resilient to failures.

Setting a retry policy for a job

To configure a retry policy, you provide a RetryPolicyConfiguration field using the StartJobRun API. An example retryPolicyConfiguration is shown here:

aws emr-containers start-job-run \
--virtual-cluster-id cluster_id \
--name sample-job-name \
--execution-role-arn execution-role-arn \
--release-label emr-6.9.0-latest \
--job-driver '{
  "sparkSubmitJobDriver": {
    "entryPoint": "local:///usr/lib/spark/examples/src/main/python/pi.py",
    "entryPointArguments": [ "2" ],
    "sparkSubmitParameters": "--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1"
  }
}' \
--retry-policy-configuration '{
    "maxAttempts": 5
  }' \
--configuration-overrides '{
  "monitoringConfiguration": {
    "cloudWatchMonitoringConfiguration": {
      "logGroupName": "my_log_group_name",
      "logStreamNamePrefix": "my_log_stream_prefix"
    },
    "s3MonitoringConfiguration": {
       "logUri": "s3://amzn-s3-demo-logging-bucket"
    }
  }
}'

Configure the maxAttempts field with the maximum number of times you want the job driver pod to be restarted if it fails or is deleted. The execution interval between two job driver retry attempts is an exponential retry interval of (10 seconds, 20 seconds, 40 seconds ...) which is capped at 6 minutes, as described in the Kubernetes documentation.

Retry policy configuration values

Retrieving a retry policy status for a job

You can view the status of the retry attempts for a job with the ListJobRuns and DescribeJobRun APIs. Once you request a job with an enabled retry policy configuration, the ListJobRun and DescribeJobRun responses will contain the status of the retry policy in the RetryPolicyExecution field. In addition, the DescribeJobRun response will contain the RetryPolicyConfiguration that was input in the StartJobRun request for the job.

Sample responses

ListJobRuns response

{
  "jobRuns": [
    ...
    ...
    "retryPolicyExecution" : {
      "currentAttemptCount": 2
    }
    ...
    ...
  ]
}

DescribeJobRun response

{
  ...
  ...
  "retryPolicyConfiguration": {
    "maxAttempts": 5
   },
   "retryPolicyExecution" : {
    "currentAttemptCount": 2
  },
  ...
  ...
}

These fields will not be visible when retry policy is disabled in the job, as described below in Retry policy configuration values.

Monitoring a job with a retry policy

When you enable a retry policy, a CloudWatch event is generated for every job driver that is created. To subscribe to these events, set up a CloudWatch event rule using the following command:

aws events put-rule \
--name cwe-test \
--event-pattern '{"detail-type": ["EMR Job Run New Driver Attempt"]}'

The event will return information on the newDriverPodName, newDriverCreatedAt timestamp, previousDriverFailureMessage, and currentAttemptCount of the job drivers. These events will not be created if the retry policy is disabled.

For more information on how to monitor your job with CloudWatch events, see Monitor jobs with Amazon CloudWatch Events.

Finding logs for drivers and executors

Driver pod names follow the format spark-<job id>-driver-<random-suffix>. The same random-suffix is added to the executor pod names that the driver spawns. When you use this random-suffix, you can find logs for a driver and its associated executors. The random-suffix is only present if the retry policy is enabled for the job; otherwise, the random-suffix is absent.

For more information on how to configure jobs with monitoring configuration for logging, see Run a Spark application.