There are more AWS SDK examples available in the [AWS Doc SDK Examples](https://github.com/awsdocs/aws-doc-sdk-examples) GitHub repo.
# Code examples for Amazon EMR using AWS SDKs
The following code examples show you how to use Amazon EMR with an AWS software development kit (SDK).
*Actions* are code excerpts from larger programs and must be run in context. While actions show you how to call individual service functions, you can see actions in context in their related scenarios.
*Scenarios* are code examples that show you how to accomplish specific tasks by calling multiple functions within a service or combined with other AWS services.
**More resources**
+ **[ Amazon EMR Management Guide](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html)** – More information about Amazon EMR.
+ **[Amazon EMR API Reference](https://docs.aws.amazon.com/emr/latest/APIReference/Welcome.html)** – Details about all available Amazon EMR actions.
+ **[AWS Developer Center](https://aws.amazon.com/developer/code-examples/?awsf.sdk-code-examples-product=product%23emr)** – Code examples that you can filter by category or full-text search.
+ **[AWS SDK Examples](https://github.com/awsdocs/aws-doc-sdk-examples)** – GitHub repo with complete code in preferred languages. Includes instructions for setting up and running the code.
**Contents**
+ [Basics](emr_code_examples_basics.md)
+ [Actions](emr_code_examples_actions.md)
+ [`AddJobFlowSteps`](emr_example_emr_AddJobFlowSteps_section.md)
+ [`DescribeCluster`](emr_example_emr_DescribeCluster_section.md)
+ [`DescribeStep`](emr_example_emr_DescribeStep_section.md)
+ [`ListSteps`](emr_example_emr_ListSteps_section.md)
+ [`RunJobFlow`](emr_example_emr_RunJobFlow_section.md)
+ [`TerminateJobFlows`](emr_example_emr_TerminateJobFlows_section.md)
+ [Scenarios](emr_code_examples_scenarios.md)
+ [Create a short-lived Amazon EMR cluster and run a step](emr_example_emr_Scenario_ShortLivedEmrCluster_section.md)
+ [Getting started with Amazon EMR](emr_example_emr_GettingStarted_037_section.md)
+ [Run a shell script to install libraries](emr_example_emr_Usage_InstallLibrariesWithSsm_section.md)
# Basic examples for Amazon EMR using AWS SDKs
The following code examples show how to use the basics of Amazon EMR with AWS SDKs.
**Contents**
+ [Actions](emr_code_examples_actions.md)
+ [`AddJobFlowSteps`](emr_example_emr_AddJobFlowSteps_section.md)
+ [`DescribeCluster`](emr_example_emr_DescribeCluster_section.md)
+ [`DescribeStep`](emr_example_emr_DescribeStep_section.md)
+ [`ListSteps`](emr_example_emr_ListSteps_section.md)
+ [`RunJobFlow`](emr_example_emr_RunJobFlow_section.md)
+ [`TerminateJobFlows`](emr_example_emr_TerminateJobFlows_section.md)
# Actions for Amazon EMR using AWS SDKs
The following code examples demonstrate how to perform individual Amazon EMR actions with AWS SDKs. Each example includes a link to GitHub, where you can find instructions for setting up and running the code.
These excerpts call the Amazon EMR API and are code excerpts from larger programs that must be run in context. You can see actions in context in [Scenarios for Amazon EMR using AWS SDKs](emr_code_examples_scenarios.md).
The following examples include only the most commonly used actions. For a complete list, see the [Amazon EMR API Reference](https://docs.aws.amazon.com/emr/latest/APIReference/Welcome.html).
**Topics**
+ [`AddJobFlowSteps`](emr_example_emr_AddJobFlowSteps_section.md)
+ [`DescribeCluster`](emr_example_emr_DescribeCluster_section.md)
+ [`DescribeStep`](emr_example_emr_DescribeStep_section.md)
+ [`ListSteps`](emr_example_emr_ListSteps_section.md)
+ [`RunJobFlow`](emr_example_emr_RunJobFlow_section.md)
+ [`TerminateJobFlows`](emr_example_emr_TerminateJobFlows_section.md)
# Use `AddJobFlowSteps` with an AWS SDK
The following code examples show how to use `AddJobFlowSteps`.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples).
Add a Spark step, which is run by the cluster as soon as it is added.
```
def add_step(cluster_id, name, script_uri, script_args, emr_client):
"""
Adds a job step to the specified cluster. This example adds a Spark
step, which is run by the cluster as soon as it is added.
:param cluster_id: The ID of the cluster.
:param name: The name of the step.
:param script_uri: The URI where the Python script is stored.
:param script_args: Arguments to pass to the Python script.
:param emr_client: The Boto3 EMR client object.
:return: The ID of the newly added step.
"""
try:
response = emr_client.add_job_flow_steps(
JobFlowId=cluster_id,
Steps=[
{
"Name": name,
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
script_uri,
*script_args,
],
},
}
],
)
step_id = response["StepIds"][0]
logger.info("Started step with ID %s", step_id)
except ClientError:
logger.exception("Couldn't start step %s with URI %s.", name, script_uri)
raise
else:
return step_id
```
Run an Amazon EMR File System (EMRFS) command as a job step on a cluster. This can be used to automate EMRFS commands on a cluster instead of running commands manually through an SSH connection.
```
import boto3
from botocore.exceptions import ClientError
def add_emrfs_step(command, bucket_url, cluster_id, emr_client):
"""
Add an EMRFS command as a job flow step to an existing cluster.
:param command: The EMRFS command to run.
:param bucket_url: The URL of a bucket that contains tracking metadata.
:param cluster_id: The ID of the cluster to update.
:param emr_client: The Boto3 Amazon EMR client object.
:return: The ID of the added job flow step. Status can be tracked by calling
the emr_client.describe_step() function.
"""
job_flow_step = {
"Name": "Example EMRFS Command Step",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": ["/usr/bin/emrfs", command, bucket_url],
},
}
try:
response = emr_client.add_job_flow_steps(
JobFlowId=cluster_id, Steps=[job_flow_step]
)
step_id = response["StepIds"][0]
print(f"Added step {step_id} to cluster {cluster_id}.")
except ClientError:
print(f"Couldn't add a step to cluster {cluster_id}.")
raise
else:
return step_id
def usage_demo():
emr_client = boto3.client("emr")
# Assumes the first waiting cluster has EMRFS enabled and has created metadata
# with the default name of 'EmrFSMetadata'.
cluster = emr_client.list_clusters(ClusterStates=["WAITING"])["Clusters"][0]
add_emrfs_step(
"sync", "s3://elasticmapreduce/samples/cloudfront", cluster["Id"], emr_client
)
if __name__ == "__main__":
usage_demo()
```
+ For API details, see [AddJobFlowSteps](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/AddJobFlowSteps) in *AWS SDK for Python (Boto3) API Reference*.
------
#### [ SAP ABAP ]
**SDK for SAP ABAP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples).
```
TRY.
" Build args list for Spark submit
DATA lt_args TYPE /aws1/cl_emrxmlstringlist_w=>tt_xmlstringlist.
APPEND NEW /aws1/cl_emrxmlstringlist_w( 'spark-submit' ) TO lt_args.
APPEND NEW /aws1/cl_emrxmlstringlist_w( '--deploy-mode' ) TO lt_args.
APPEND NEW /aws1/cl_emrxmlstringlist_w( 'cluster' ) TO lt_args.
APPEND NEW /aws1/cl_emrxmlstringlist_w( iv_script_uri ) TO lt_args.
APPEND LINES OF it_script_args TO lt_args.
" Create step configuration
DATA(lo_hadoop_jar_step) = NEW /aws1/cl_emrhadoopjarstepcfg(
iv_jar = 'command-runner.jar'
it_args = lt_args
).
DATA(lo_step_config) = NEW /aws1/cl_emrstepconfig(
iv_name = iv_name
iv_actiononfailure = 'CONTINUE'
io_hadoopjarstep = lo_hadoop_jar_step
).
DATA lt_steps TYPE /aws1/cl_emrstepconfig=>tt_stepconfiglist.
APPEND lo_step_config TO lt_steps.
DATA(lo_result) = lo_emr->addjobflowsteps(
iv_jobflowid = iv_cluster_id
it_steps = lt_steps
).
" Get first step ID
DATA(lt_step_ids) = lo_result->get_stepids( ).
READ TABLE lt_step_ids INDEX 1 INTO DATA(lo_step_id_obj).
IF sy-subrc = 0.
ov_step_id = lo_step_id_obj->get_value( ).
MESSAGE |Step added with ID { ov_step_id }| TYPE 'I'.
ENDIF.
CATCH /aws1/cx_emrinternalservererr INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ For API details, see [AddJobFlowSteps](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html) in *AWS SDK for SAP ABAP API reference*.
------
# Use `DescribeCluster` with an AWS SDK or CLI
The following code examples show how to use `DescribeCluster`.
Action examples are code excerpts from larger programs and must be run in context. You can see this action in context in the following code example:
+ [Getting started with Amazon EMR](emr_example_emr_GettingStarted_037_section.md)
------
#### [ CLI ]
**AWS CLI**
Command:
```
aws emr describe-cluster --cluster-id j-XXXXXXXX
```
Output:
```
For release-label based uniform instance groups cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1436475075.199,
"CreationDateTime": 1436474656.563,
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"ServiceAccessSecurityGroup": "sg-xxxxxxxx",
"EmrManagedMasterSecurityGroup": "sg-xxxxxxxx",
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2KeyName": "myKey",
"Ec2AvailabilityZone": "us-east-1c",
"EmrManagedSlaveSecurityGroup": "sg-yyyyyyyyy"
},
"Name": "My Cluster",
"ServiceRole": "EMR_DefaultRole",
"Tags": [],
"TerminationProtected": true,
"UnhealthyNodeReplacement": true,
"ReleaseLabel": "emr-4.0.0",
"NormalizedInstanceHours": 96,
"InstanceGroups": [
{
"RequestedInstanceCount": 2,
"Status": {
"Timeline": {
"ReadyDateTime": 1436475074.245,
"CreationDateTime": 1436474656.564,
"EndDateTime": 1436638158.387
},
"State": "RUNNING",
"StateChangeReason": {
"Message": "",
}
},
"Name": "CORE",
"InstanceGroupType": "CORE",
"Id": "ig-YYYYYYY",
"Configurations": [],
"InstanceType": "m3.large",
"Market": "ON_DEMAND",
"RunningInstanceCount": 2
},
{
"RequestedInstanceCount": 1,
"Status": {
"Timeline": {
"ReadyDateTime": 1436475074.245,
"CreationDateTime": 1436474656.564,
"EndDateTime": 1436638158.387
},
"State": "RUNNING",
"StateChangeReason": {
"Message": "",
}
},
"Name": "MASTER",
"InstanceGroupType": "MASTER",
"Id": "ig-XXXXXXXXX",
"Configurations": [],
"InstanceType": "m3.large",
"Market": "ON_DEMAND",
"RunningInstanceCount": 1
}
],
"Applications": [
{
"Name": "Hadoop"
}
],
"VisibleToAllUsers": true,
"BootstrapActions": [],
"MasterPublicDnsName": "ec2-54-147-144-78.compute-1.amazonaws.com",
"AutoTerminate": false,
"Id": "j-XXXXXXXX",
"Configurations": [
{
"Properties": {
"fs.s3.consistent.retryPeriodSeconds": "20",
"fs.s3.enableServerSideEncryption": "true",
"fs.s3.consistent": "false",
"fs.s3.consistent.retryCount": "2"
},
"Classification": "emrfs-site"
}
]
}
}
For release-label based instance fleet cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1487897289.705,
"CreationDateTime": 1487896933.942
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"EmrManagedMasterSecurityGroup": "sg-xxxxx",
"RequestedEc2AvailabilityZones": [],
"RequestedEc2SubnetIds": [],
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2AvailabilityZone": "us-east-1a",
"EmrManagedSlaveSecurityGroup": "sg-xxxxx"
},
"Name": "My Cluster",
"ServiceRole": "EMR_DefaultRole",
"Tags": [],
"TerminationProtected": false,
"UnhealthyNodeReplacement": false,
"ReleaseLabel": "emr-5.2.0",
"NormalizedInstanceHours": 472,
"InstanceCollectionType": "INSTANCE_FLEET",
"InstanceFleets": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1487897212.74,
"CreationDateTime": 1487896933.948
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"ProvisionedSpotCapacity": 1,
"Name": "MASTER",
"InstanceFleetType": "MASTER",
"LaunchSpecifications": {
"SpotSpecification": {
"TimeoutDurationMinutes": 60,
"TimeoutAction": "TERMINATE_CLUSTER"
}
},
"TargetSpotCapacity": 1,
"ProvisionedOnDemandCapacity": 0,
"InstanceTypeSpecifications": [
{
"BidPrice": "0.5",
"InstanceType": "m3.xlarge",
"WeightedCapacity": 1
}
],
"Id": "if-xxxxxxx",
"TargetOnDemandCapacity": 0
}
],
"Applications": [
{
"Version": "2.7.3",
"Name": "Hadoop"
}
],
"ScaleDownBehavior": "TERMINATE_AT_INSTANCE_HOUR",
"VisibleToAllUsers": true,
"BootstrapActions": [],
"MasterPublicDnsName": "ec2-xxx-xx-xxx-xx.compute-1.amazonaws.com",
"AutoTerminate": false,
"Id": "j-xxxxx",
"Configurations": []
}
}
For ami based uniform instance group cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1399400564.432,
"CreationDateTime": 1399400268.62
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2AvailabilityZone": "us-east-1c"
},
"Name": "My Cluster",
"Tags": [],
"TerminationProtected": true,
"UnhealthyNodeReplacement": true,
"RunningAmiVersion": "2.5.4",
"InstanceGroups": [
{
"RequestedInstanceCount": 1,
"Status": {
"Timeline": {
"ReadyDateTime": 1399400558.848,
"CreationDateTime": 1399400268.621
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"Name": "Master instance group",
"InstanceGroupType": "MASTER",
"InstanceType": "m1.small",
"Id": "ig-ABCD",
"Market": "ON_DEMAND",
"RunningInstanceCount": 1
},
{
"RequestedInstanceCount": 2,
"Status": {
"Timeline": {
"ReadyDateTime": 1399400564.439,
"CreationDateTime": 1399400268.621
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"Name": "Core instance group",
"InstanceGroupType": "CORE",
"InstanceType": "m1.small",
"Id": "ig-DEF",
"Market": "ON_DEMAND",
"RunningInstanceCount": 2
}
],
"Applications": [
{
"Version": "1.0.3",
"Name": "hadoop"
}
],
"BootstrapActions": [],
"VisibleToAllUsers": false,
"RequestedAmiVersion": "2.4.2",
"LogUri": "s3://myLogUri/",
"AutoTerminate": false,
"Id": "j-XXXXXXXX"
}
}
```
+ For API details, see [DescribeCluster](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/describe-cluster.html) in *AWS CLI Command Reference*.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples).
```
def describe_cluster(cluster_id, emr_client):
"""
Gets detailed information about a cluster.
:param cluster_id: The ID of the cluster to describe.
:param emr_client: The Boto3 EMR client object.
:return: The retrieved cluster information.
"""
try:
response = emr_client.describe_cluster(ClusterId=cluster_id)
cluster = response["Cluster"]
logger.info("Got data for cluster %s.", cluster["Name"])
except ClientError:
logger.exception("Couldn't get data for cluster %s.", cluster_id)
raise
else:
return cluster
```
+ For API details, see [DescribeCluster](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/DescribeCluster) in *AWS SDK for Python (Boto3) API Reference*.
------
#### [ SAP ABAP ]
**SDK for SAP ABAP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples).
```
TRY.
oo_result = lo_emr->describecluster(
iv_clusterid = iv_cluster_id
).
DATA(lo_cluster) = oo_result->get_cluster( ).
DATA(lv_cluster_name) = lo_cluster->get_name( ).
MESSAGE |Retrieved cluster information for { lv_cluster_name }| TYPE 'I'.
CATCH /aws1/cx_emrinternalserverex INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrinvalidrequestex INTO DATA(lo_invalid_error).
lv_error = lo_invalid_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ For API details, see [DescribeCluster](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html) in *AWS SDK for SAP ABAP API reference*.
------
# Use `DescribeStep` with an AWS SDK or CLI
The following code examples show how to use `DescribeStep`.
Action examples are code excerpts from larger programs and must be run in context. You can see this action in context in the following code example:
+ [Getting started with Amazon EMR](emr_example_emr_GettingStarted_037_section.md)
------
#### [ CLI ]
**AWS CLI**
The following command describes a step with the step ID `s-3LZC0QUT43AM` in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr describe-step --cluster-id j-3SD91U2E1L2QX --step-id s-3LZC0QUT43AM
```
Output:
```
{
"Step": {
"Status": {
"Timeline": {
"EndDateTime": 1433200470.481,
"CreationDateTime": 1433199926.597,
"StartDateTime": 1433200404.959
},
"State": "COMPLETED",
"StateChangeReason": {}
},
"Config": {
"Args": [
"s3://us-west-2.elasticmapreduce/libs/hive/hive-script",
"--base-path",
"s3://us-west-2.elasticmapreduce/libs/hive/",
"--install-hive",
"--hive-versions",
"0.13.1"
],
"Jar": "s3://us-west-2.elasticmapreduce/libs/script-runner/script-runner.jar",
"Properties": {}
},
"Id": "s-3LZC0QUT43AM",
"ActionOnFailure": "TERMINATE_CLUSTER",
"Name": "Setup hive"
}
}
```
+ For API details, see [DescribeStep](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/describe-step.html) in *AWS CLI Command Reference*.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples).
```
def describe_step(cluster_id, step_id, emr_client):
"""
Gets detailed information about the specified step, including the current state of
the step.
:param cluster_id: The ID of the cluster.
:param step_id: The ID of the step.
:param emr_client: The Boto3 EMR client object.
:return: The retrieved information about the specified step.
"""
try:
response = emr_client.describe_step(ClusterId=cluster_id, StepId=step_id)
step = response["Step"]
logger.info("Got data for step %s.", step_id)
except ClientError:
logger.exception("Couldn't get data for step %s.", step_id)
raise
else:
return step
```
+ For API details, see [DescribeStep](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/DescribeStep) in *AWS SDK for Python (Boto3) API Reference*.
------
#### [ SAP ABAP ]
**SDK for SAP ABAP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples).
```
TRY.
oo_result = lo_emr->describestep(
iv_clusterid = iv_cluster_id
iv_stepid = iv_step_id
).
DATA(lo_step) = oo_result->get_step( ).
DATA(lv_step_name) = lo_step->get_name( ).
MESSAGE |Retrieved step information for { lv_step_name }| TYPE 'I'.
CATCH /aws1/cx_emrinternalserverex INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrinvalidrequestex INTO DATA(lo_invalid_error).
lv_error = lo_invalid_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ For API details, see [DescribeStep](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html) in *AWS SDK for SAP ABAP API reference*.
------
# Use `ListSteps` with an AWS SDK or CLI
The following code examples show how to use `ListSteps`.
------
#### [ CLI ]
**AWS CLI**
The following command lists all of the steps in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr list-steps --cluster-id j-3SD91U2E1L2QX
```
+ For API details, see [ListSteps](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-steps.html) in *AWS CLI Command Reference*.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples).
```
def list_steps(cluster_id, emr_client):
"""
Gets a list of steps for the specified cluster. In this example, all steps are
returned, including completed and failed steps.
:param cluster_id: The ID of the cluster.
:param emr_client: The Boto3 EMR client object.
:return: The list of steps for the specified cluster.
"""
try:
response = emr_client.list_steps(ClusterId=cluster_id)
steps = response["Steps"]
logger.info("Got %s steps for cluster %s.", len(steps), cluster_id)
except ClientError:
logger.exception("Couldn't get steps for cluster %s.", cluster_id)
raise
else:
return steps
```
+ For API details, see [ListSteps](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/ListSteps) in *AWS SDK for Python (Boto3) API Reference*.
------
#### [ SAP ABAP ]
**SDK for SAP ABAP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples).
```
TRY.
oo_result = lo_emr->liststeps(
iv_clusterid = iv_cluster_id
).
DATA(lt_steps) = oo_result->get_steps( ).
DATA(lv_step_count) = lines( lt_steps ).
MESSAGE |Retrieved { lv_step_count } steps for cluster| TYPE 'I'.
CATCH /aws1/cx_emrinternalserverex INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrinvalidrequestex INTO DATA(lo_invalid_error).
lv_error = lo_invalid_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ For API details, see [ListSteps](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html) in *AWS SDK for SAP ABAP API reference*.
------
# Use `RunJobFlow` with an AWS SDK
The following code examples show how to use `RunJobFlow`.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples).
```
def run_job_flow(
name,
log_uri,
keep_alive,
applications,
job_flow_role,
service_role,
security_groups,
steps,
emr_client,
):
"""
Runs a job flow with the specified steps. A job flow creates a cluster of
instances and adds steps to be run on the cluster. Steps added to the cluster
are run as soon as the cluster is ready.
This example uses the 'emr-5.30.1' release. A list of recent releases can be
found here:
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html.
:param name: The name of the cluster.
:param log_uri: The URI where logs are stored. This can be an Amazon S3 bucket URL,
such as 's3://my-log-bucket'.
:param keep_alive: When True, the cluster is put into a Waiting state after all
steps are run. When False, the cluster terminates itself when
the step queue is empty.
:param applications: The applications to install on each instance in the cluster,
such as Hive or Spark.
:param job_flow_role: The IAM role assumed by the cluster.
:param service_role: The IAM role assumed by the service.
:param security_groups: The security groups to assign to the cluster instances.
Amazon EMR adds all needed rules to these groups, so
they can be empty if you require only the default rules.
:param steps: The job flow steps to add to the cluster. These are run in order
when the cluster is ready.
:param emr_client: The Boto3 EMR client object.
:return: The ID of the newly created cluster.
"""
try:
response = emr_client.run_job_flow(
Name=name,
LogUri=log_uri,
ReleaseLabel="emr-5.30.1",
Instances={
"MasterInstanceType": "m5.xlarge",
"SlaveInstanceType": "m5.xlarge",
"InstanceCount": 3,
"KeepJobFlowAliveWhenNoSteps": keep_alive,
"EmrManagedMasterSecurityGroup": security_groups["manager"].id,
"EmrManagedSlaveSecurityGroup": security_groups["worker"].id,
},
Steps=[
{
"Name": step["name"],
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
step["script_uri"],
*step["script_args"],
],
},
}
for step in steps
],
Applications=[{"Name": app} for app in applications],
JobFlowRole=job_flow_role.name,
ServiceRole=service_role.name,
EbsRootVolumeSize=10,
VisibleToAllUsers=True,
)
cluster_id = response["JobFlowId"]
logger.info("Created cluster %s.", cluster_id)
except ClientError:
logger.exception("Couldn't create cluster.")
raise
else:
return cluster_id
```
+ For API details, see [RunJobFlow](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/RunJobFlow) in *AWS SDK for Python (Boto3) API Reference*.
------
#### [ SAP ABAP ]
**SDK for SAP ABAP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples).
```
TRY.
" Create instances configuration
DATA(lo_instances) = NEW /aws1/cl_emrjobflowinstsconfig(
iv_masterinstancetype = 'm5.xlarge'
iv_slaveinstancetype = 'm5.xlarge'
iv_instancecount = 3
iv_keepjobflowalivewhennos00 = iv_keep_alive
iv_emrmanagedmastersecgroup = iv_primary_sec_grp
iv_emrmanagedslavesecgroup = iv_secondary_sec_grp
).
DATA(lo_result) = lo_emr->runjobflow(
iv_name = iv_name
iv_loguri = iv_log_uri
iv_releaselabel = 'emr-5.30.1'
io_instances = lo_instances
it_steps = it_steps
it_applications = it_applications
iv_jobflowrole = iv_job_flow_role
iv_servicerole = iv_service_role
iv_ebsrootvolumesize = 10
iv_visibletoallusers = abap_true
).
ov_cluster_id = lo_result->get_jobflowid( ).
MESSAGE 'EMR cluster created successfully.' TYPE 'I'.
CATCH /aws1/cx_emrinternalservererr INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
CATCH /aws1/cx_emrclientexc INTO DATA(lo_client_error).
lv_error = lo_client_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ For API details, see [RunJobFlow](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html) in *AWS SDK for SAP ABAP API reference*.
------
# Use `TerminateJobFlows` with an AWS SDK
The following code examples show how to use `TerminateJobFlows`.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples).
```
def terminate_cluster(cluster_id, emr_client):
"""
Terminates a cluster. This terminates all instances in the cluster and cannot
be undone. Any data not saved elsewhere, such as in an Amazon S3 bucket, is lost.
:param cluster_id: The ID of the cluster to terminate.
:param emr_client: The Boto3 EMR client object.
"""
try:
emr_client.terminate_job_flows(JobFlowIds=[cluster_id])
logger.info("Terminated cluster %s.", cluster_id)
except ClientError:
logger.exception("Couldn't terminate cluster %s.", cluster_id)
raise
```
+ For API details, see [TerminateJobFlows](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/TerminateJobFlows) in *AWS SDK for Python (Boto3) API Reference*.
------
#### [ SAP ABAP ]
**SDK for SAP ABAP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/emr#code-examples).
```
TRY.
DATA lt_cluster_ids TYPE /aws1/cl_emrxmlstringlist_w=>tt_xmlstringlist.
APPEND NEW /aws1/cl_emrxmlstringlist_w( iv_cluster_id ) TO lt_cluster_ids.
lo_emr->terminatejobflows(
it_jobflowids = lt_cluster_ids
).
MESSAGE 'EMR cluster terminated successfully.' TYPE 'I'.
CATCH /aws1/cx_emrinternalservererr INTO DATA(lo_internal_error).
DATA(lv_error) = lo_internal_error->if_message~get_text( ).
MESSAGE lv_error TYPE 'E'.
ENDTRY.
```
+ For API details, see [TerminateJobFlows](https://docs.aws.amazon.com/sdk-for-sap-abap/v1/api/latest/index.html) in *AWS SDK for SAP ABAP API reference*.
------
# Scenarios for Amazon EMR using AWS SDKs
The following code examples show you how to implement common scenarios in Amazon EMR with AWS SDKs. These scenarios show you how to accomplish specific tasks by calling multiple functions within Amazon EMR or combined with other AWS services. Each scenario includes a link to the complete source code, where you can find instructions on how to set up and run the code.
Scenarios target an intermediate level of experience to help you understand service actions in context.
**Topics**
+ [Create a short-lived Amazon EMR cluster and run a step](emr_example_emr_Scenario_ShortLivedEmrCluster_section.md)
+ [Getting started with Amazon EMR](emr_example_emr_GettingStarted_037_section.md)
+ [Run a shell script to install libraries](emr_example_emr_Usage_InstallLibrariesWithSsm_section.md)
# Create a short-lived Amazon EMR cluster and run a step using an AWS SDK
The following code example shows how to create a short-lived Amazon EMR cluster that runs a step and automatically terminates after the step completes.
------
#### [ Python ]
**SDK for Python (Boto3)**
Create a short-lived Amazon EMR cluster that estimates the value of pi using Apache Spark to parallelize a large number of calculations. The job writes output to Amazon EMR logs and to an Amazon Simple Storage Service (Amazon S3) bucket. The cluster terminates itself after completing the job.
+ Create an Amazon S3 bucket and upload a job script.
+ Create AWS Identity and Access Management (IAM) roles.
+ Create Amazon Elastic Compute Cloud (Amazon EC2) security groups.
+ Create a short-lived cluster and run a single job step.
This example is best viewed on GitHub. For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr).
**Services used in this example**
+ Amazon EMR
------
# Getting started with Amazon EMR
The following code example shows how to:
+ Create an EC2 key pair
+ Set up storage and prepare your application
+ Clean up resources
------
#### [ Bash ]
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/037-emr-gs) repository.
```
#!/bin/bash
# EMR Getting Started Tutorial Script
# This script automates the steps in the Amazon EMR Getting Started tutorial
# Set up logging
LOG_FILE="emr-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "Starting Amazon EMR Getting Started Tutorial Script"
echo "Logging to $LOG_FILE"
# Function to handle errors
handle_error() {
echo "ERROR: $1"
echo "Resources created so far:"
if [ -n "$BUCKET_NAME" ]; then echo "- S3 Bucket: $BUCKET_NAME"; fi
if [ -n "$CLUSTER_ID" ]; then echo "- EMR Cluster: $CLUSTER_ID"; fi
echo "Attempting to clean up resources..."
cleanup
exit 1
}
# Function to clean up resources
cleanup() {
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "${CLEANUP_CHOICE,,}" == "y" ]]; then
echo "Starting cleanup process..."
# Terminate EMR cluster if it exists
if [ -n "$CLUSTER_ID" ]; then
echo "Terminating EMR cluster: $CLUSTER_ID"
aws emr terminate-clusters --cluster-ids "$CLUSTER_ID"
echo "Waiting for cluster to terminate..."
aws emr wait cluster-terminated --cluster-id "$CLUSTER_ID"
echo "Cluster terminated successfully."
fi
# Delete S3 bucket and contents if it exists
if [ -n "$BUCKET_NAME" ]; then
echo "Deleting S3 bucket contents: $BUCKET_NAME"
aws s3 rm "s3://$BUCKET_NAME" --recursive
echo "Deleting S3 bucket: $BUCKET_NAME"
aws s3 rb "s3://$BUCKET_NAME"
fi
echo "Cleanup completed."
else
echo "Cleanup skipped. Resources will remain in your AWS account."
echo "To avoid ongoing charges, remember to manually delete these resources."
fi
}
# Generate a random identifier for S3 bucket
RANDOM_ID=$(openssl rand -hex 6)
BUCKET_NAME="emr${RANDOM_ID}"
echo "Using bucket name: $BUCKET_NAME"
# Create S3 bucket
echo "Creating S3 bucket: $BUCKET_NAME"
aws s3 mb "s3://$BUCKET_NAME" || handle_error "Failed to create S3 bucket"
echo "S3 bucket created successfully."
# Create PySpark script
echo "Creating PySpark script: health_violations.py"
cat > health_violations.py << 'EOL'
import argparse
from pyspark.sql import SparkSession
def calculate_red_violations(data_source, output_uri):
"""
Processes sample food establishment inspection data and queries the data to find the top 10 establishments
with the most Red violations from 2006 to 2020.
:param data_source: The URI of your food establishment data CSV, such as 's3://emr-tutorial-bucket/food-establishment-data.csv'.
:param output_uri: The URI where output is written, such as 's3://emr-tutorial-bucket/restaurant_violation_results'.
"""
with SparkSession.builder.appName("Calculate Red Health Violations").getOrCreate() as spark:
# Load the restaurant violation CSV data
if data_source is not None:
restaurants_df = spark.read.option("header", "true").csv(data_source)
# Create an in-memory DataFrame to query
restaurants_df.createOrReplaceTempView("restaurant_violations")
# Create a DataFrame of the top 10 restaurants with the most Red violations
top_red_violation_restaurants = spark.sql("""SELECT name, count(*) AS total_red_violations
FROM restaurant_violations
WHERE violation_type = 'RED'
GROUP BY name
ORDER BY total_red_violations DESC LIMIT 10""")
# Write the results to the specified output URI
top_red_violation_restaurants.write.option("header", "true").mode("overwrite").csv(output_uri)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
'--data_source', help="The URI for you CSV restaurant data, like an S3 bucket location.")
parser.add_argument(
'--output_uri', help="The URI where output is saved, like an S3 bucket location.")
args = parser.parse_args()
calculate_red_violations(args.data_source, args.output_uri)
EOL
# Upload PySpark script to S3
echo "Uploading PySpark script to S3"
aws s3 cp health_violations.py "s3://$BUCKET_NAME/" || handle_error "Failed to upload PySpark script"
echo "PySpark script uploaded successfully."
# Download and prepare sample data
echo "Downloading sample data"
curl -o food_establishment_data.zip https://docs.aws.amazon.com/emr/latest/ManagementGuide/samples/food_establishment_data.zip || handle_error "Failed to download sample data"
unzip -o food_establishment_data.zip || handle_error "Failed to unzip sample data"
echo "Sample data downloaded and extracted successfully."
# Upload sample data to S3
echo "Uploading sample data to S3"
aws s3 cp food_establishment_data.csv "s3://$BUCKET_NAME/" || handle_error "Failed to upload sample data"
echo "Sample data uploaded successfully."
# Create IAM default roles for EMR
echo "Creating IAM default roles for EMR"
aws emr create-default-roles || handle_error "Failed to create default roles"
echo "IAM default roles created successfully."
# Check if EC2 key pair exists
echo "Checking for EC2 key pair"
KEY_PAIRS=$(aws ec2 describe-key-pairs --query "KeyPairs[*].KeyName" --output text)
if [ -z "$KEY_PAIRS" ]; then
echo "No EC2 key pairs found. Creating a new key pair..."
KEY_NAME="emr-tutorial-key-$RANDOM_ID"
aws ec2 create-key-pair --key-name "$KEY_NAME" --query "KeyMaterial" --output text > "${KEY_NAME}.pem"
chmod 400 "${KEY_NAME}.pem"
echo "Created new key pair: $KEY_NAME"
else
# Use the first available key pair
KEY_NAME=$(echo "$KEY_PAIRS" | awk '{print $1}')
echo "Using existing key pair: $KEY_NAME"
fi
# Launch EMR cluster
echo "Launching EMR cluster with Spark"
CLUSTER_RESPONSE=$(aws emr create-cluster \
--name "EMR Tutorial Cluster" \
--release-label emr-6.10.0 \
--applications Name=Spark \
--ec2-attributes KeyName="$KEY_NAME" \
--instance-type m5.xlarge \
--instance-count 3 \
--use-default-roles \
--log-uri "s3://$BUCKET_NAME/logs/")
# Check for errors in the response
if echo "$CLUSTER_RESPONSE" | grep -i "error" > /dev/null; then
handle_error "Failed to create EMR cluster: $CLUSTER_RESPONSE"
fi
# Extract cluster ID
CLUSTER_ID=$(echo "$CLUSTER_RESPONSE" | grep -o '"ClusterId": "[^"]*' | cut -d'"' -f4)
if [ -z "$CLUSTER_ID" ]; then
handle_error "Failed to extract cluster ID from response"
fi
echo "EMR cluster created with ID: $CLUSTER_ID"
# Wait for cluster to be ready
echo "Waiting for cluster to be ready (this may take several minutes)..."
aws emr wait cluster-running --cluster-id "$CLUSTER_ID" || handle_error "Cluster failed to reach running state"
# Check if cluster is in WAITING state
CLUSTER_STATE=$(aws emr describe-cluster --cluster-id "$CLUSTER_ID" --query "Cluster.Status.State" --output text)
if [ "$CLUSTER_STATE" != "WAITING" ]; then
echo "Waiting for cluster to reach WAITING state..."
while [ "$CLUSTER_STATE" != "WAITING" ]; do
sleep 30
CLUSTER_STATE=$(aws emr describe-cluster --cluster-id "$CLUSTER_ID" --query "Cluster.Status.State" --output text)
echo "Current cluster state: $CLUSTER_STATE"
# Check for error states
if [[ "$CLUSTER_STATE" == "TERMINATED_WITH_ERRORS" || "$CLUSTER_STATE" == "TERMINATED" ]]; then
handle_error "Cluster entered error state: $CLUSTER_STATE"
fi
done
fi
echo "Cluster is now in WAITING state and ready to accept work."
# Submit Spark application as a step
echo "Submitting Spark application as a step"
STEP_RESPONSE=$(aws emr add-steps \
--cluster-id "$CLUSTER_ID" \
--steps Type=Spark,Name="Health Violations Analysis",ActionOnFailure=CONTINUE,Args=["s3://$BUCKET_NAME/health_violations.py","--data_source","s3://$BUCKET_NAME/food_establishment_data.csv","--output_uri","s3://$BUCKET_NAME/results/"])
# Check for errors in the response
if echo "$STEP_RESPONSE" | grep -i "error" > /dev/null; then
handle_error "Failed to submit step: $STEP_RESPONSE"
fi
# FIXED: Check if jq is available before using it
# Extract step ID using the appropriate method based on available tools
if command -v jq &> /dev/null; then
# Use jq if available
echo "Using jq to parse JSON response"
STEP_ID=$(echo "$STEP_RESPONSE" | jq -r '.StepIds[0]')
else
# Fallback to grep/awk if jq is not available
echo "jq not found, using grep for parsing"
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*"' | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
# Another fallback method
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*' | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
# One more attempt with a different pattern
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
echo "Full step response: $STEP_RESPONSE"
handle_error "Failed to extract step ID from response"
fi
fi
fi
fi
if [ -z "$STEP_ID" ] || [ "$STEP_ID" == "null" ]; then
echo "Full step response: $STEP_RESPONSE"
handle_error "Failed to extract valid step ID from response"
fi
echo "Step submitted with ID: $STEP_ID"
# Wait for step to complete
echo "Waiting for step to complete (this may take several minutes)..."
aws emr wait step-complete --cluster-id "$CLUSTER_ID" --step-id "$STEP_ID" || handle_error "Step failed to complete"
# Check step status
STEP_STATE=$(aws emr describe-step --cluster-id "$CLUSTER_ID" --step-id "$STEP_ID" --query "Step.Status.State" --output text)
if [ "$STEP_STATE" != "COMPLETED" ]; then
handle_error "Step did not complete successfully. Final state: $STEP_STATE"
fi
echo "Step completed successfully."
# View results
echo "Listing output files in S3"
aws s3 ls "s3://$BUCKET_NAME/results/" || handle_error "Failed to list output files"
# Download results
echo "Downloading results file"
RESULT_FILE=$(aws s3 ls "s3://$BUCKET_NAME/results/" | grep -o "part-[0-9]*.csv" | head -1)
if [ -z "$RESULT_FILE" ]; then
echo "No result file found with pattern 'part-[0-9]*.csv'. Trying to find any CSV file..."
RESULT_FILE=$(aws s3 ls "s3://$BUCKET_NAME/results/" | grep -o "part-.*\.csv" | head -1)
if [ -z "$RESULT_FILE" ]; then
echo "Listing all files in results directory:"
aws s3 ls "s3://$BUCKET_NAME/results/"
handle_error "No result file found in the output directory"
fi
fi
aws s3 cp "s3://$BUCKET_NAME/results/$RESULT_FILE" ./results.csv || handle_error "Failed to download results file"
echo "Results downloaded to results.csv"
echo "Top 10 establishments with the most red violations:"
cat results.csv
# Display SSH connection information
echo ""
echo "To connect to the cluster via SSH, use the following command:"
echo "aws emr ssh --cluster-id $CLUSTER_ID --key-pair-file ${KEY_NAME}.pem"
# Display summary of created resources
echo ""
echo "==========================================="
echo "RESOURCES CREATED"
echo "==========================================="
echo "- S3 Bucket: $BUCKET_NAME"
echo "- EMR Cluster: $CLUSTER_ID"
echo "- Results file: results.csv"
if [ -f "${KEY_NAME}.pem" ]; then
echo "- EC2 Key Pair: $KEY_NAME (saved to ${KEY_NAME}.pem)"
fi
# Offer to clean up resources
cleanup
echo "Script completed successfully."
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AddSteps](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/AddSteps)
+ [Cp](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Cp)
+ [CreateCluster](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/CreateCluster)
+ [CreateDefaultRoles](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/CreateDefaultRoles)
+ [CreateKeyPair](https://docs.aws.amazon.com/goto/aws-cli/ec2-2016-11-15/CreateKeyPair)
+ [DescribeCluster](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/DescribeCluster)
+ [DescribeKeyPairs](https://docs.aws.amazon.com/goto/aws-cli/ec2-2016-11-15/DescribeKeyPairs)
+ [DescribeStep](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/DescribeStep)
+ [Ls](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Ls)
+ [Mb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Mb)
+ [Rb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rb)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
+ [Ssh](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/Ssh)
+ [TerminateClusters](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/TerminateClusters)
+ [Wait](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/Wait)
------
# Run a shell script to install libraries on Amazon EMR instances using an AWS SDK
The following code example shows how to use AWS Systems Manager to run a shell script on Amazon EMR instances that installs additional libraries. This way, you can automate instance management instead of running commands manually through an SSH connection.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/emr#code-examples).
```
import argparse
import time
import boto3
def install_libraries_on_core_nodes(cluster_id, script_path, emr_client, ssm_client):
"""
Copies and runs a shell script on the core nodes in the cluster.
:param cluster_id: The ID of the cluster.
:param script_path: The path to the script, typically an Amazon S3 object URL.
:param emr_client: The Boto3 Amazon EMR client.
:param ssm_client: The Boto3 AWS Systems Manager client.
"""
core_nodes = emr_client.list_instances(
ClusterId=cluster_id, InstanceGroupTypes=["CORE"]
)["Instances"]
core_instance_ids = [node["Ec2InstanceId"] for node in core_nodes]
print(f"Found core instances: {core_instance_ids}.")
commands = [
# Copy the shell script from Amazon S3 to each node instance.
f"aws s3 cp {script_path} /home/hadoop",
# Run the shell script to install libraries on each node instance.
"bash /home/hadoop/install_libraries.sh",
]
for command in commands:
print(f"Sending '{command}' to core instances...")
command_id = ssm_client.send_command(
InstanceIds=core_instance_ids,
DocumentName="AWS-RunShellScript",
Parameters={"commands": [command]},
TimeoutSeconds=3600,
)["Command"]["CommandId"]
while True:
# Verify the previous step succeeded before running the next step.
cmd_result = ssm_client.list_commands(CommandId=command_id)["Commands"][0]
if cmd_result["StatusDetails"] == "Success":
print(f"Command succeeded.")
break
elif cmd_result["StatusDetails"] in ["Pending", "InProgress"]:
print(f"Command status is {cmd_result['StatusDetails']}, waiting...")
time.sleep(10)
else:
print(f"Command status is {cmd_result['StatusDetails']}, quitting.")
raise RuntimeError(
f"Command {command} failed to run. "
f"Details: {cmd_result['StatusDetails']}"
)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("cluster_id", help="The ID of the cluster.")
parser.add_argument("script_path", help="The path to the script in Amazon S3.")
args = parser.parse_args()
emr_client = boto3.client("emr")
ssm_client = boto3.client("ssm")
install_libraries_on_core_nodes(
args.cluster_id, args.script_path, emr_client, ssm_client
)
if __name__ == "__main__":
main()
```
+ For API details, see [ListInstances](https://docs.aws.amazon.com/goto/boto3/elasticmapreduce-2009-03-31/ListInstances) in *AWS SDK for Python (Boto3) API Reference*.
------