There are more AWS SDK examples available in the [AWS Doc SDK Examples](https://github.com/awsdocs/aws-doc-sdk-examples) GitHub repo.
# Amazon EMR examples using AWS CLI
The following code examples show you how to perform actions and implement common scenarios by using the AWS Command Line Interface with Amazon EMR.
*Actions* are code excerpts from larger programs and must be run in context. While actions show you how to call individual service functions, you can see actions in context in their related scenarios.
Each example includes a link to the complete source code, where you can find instructions on how to set up and run the code in context.
**Topics**
+ [Actions](#actions)
## Actions
### `add-instance-fleet`
The following code example shows how to use `add-instance-fleet`.
**AWS CLI**
**To add a task instance fleet to a cluster**
This example adds a new task instance fleet to the cluster specified.
Command:
```
aws emr add-instance-fleet --cluster-id 'j-12ABCDEFGHI34JK' --instance-fleet InstanceFleetType=TASK,TargetSpotCapacity=1,LaunchSpecifications={SpotSpecification='{TimeoutDurationMinutes=20,TimeoutAction=TERMINATE_CLUSTER}'},InstanceTypeConfigs=['{InstanceType=m3.xlarge,BidPrice=0.5}']
```
Output:
```
{
"ClusterId": "j-12ABCDEFGHI34JK",
"InstanceFleetId": "if-23ABCDEFGHI45JJ"
}
```
+ For API details, see [AddInstanceFleet](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/add-instance-fleet.html) in *AWS CLI Command Reference*.
### `add-steps`
The following code example shows how to use `add-steps`.
**AWS CLI**
**1. To add Custom JAR steps to a cluster**
Command:
```
aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,Args=arg1,arg2,arg3 Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,MainClass=mymainclass,Args=arg1,arg2,arg3
```
Required parameters:
```
Jar
```
Optional parameters:
```
Type, Name, ActionOnFailure, Args
```
Output:
```
{
"StepIds":[
"s-XXXXXXXX",
"s-YYYYYYYY"
]
}
```
**2. To add Streaming steps to a cluster**
Command:
```
aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=STREAMING,Name='Streaming Program',ActionOnFailure=CONTINUE,Args=[-files,s3://elasticmapreduce/samples/wordcount/wordSplitter.py,-mapper,wordSplitter.py,-reducer,aggregate,-input,s3://elasticmapreduce/samples/wordcount/input,-output,s3://amzn-s3-demo-bucket/wordcount/output]
```
Required parameters:
```
Type, Args
```
Optional parameters:
```
Name, ActionOnFailure
```
JSON equivalent (contents of step.json):
```
[
{
"Name": "JSON Streaming Step",
"Args": ["-files","s3://elasticmapreduce/samples/wordcount/wordSplitter.py","-mapper","wordSplitter.py","-reducer","aggregate","-input","s3://elasticmapreduce/samples/wordcount/input","-output","s3://amzn-s3-demo-bucket/wordcount/output"],
"ActionOnFailure": "CONTINUE",
"Type": "STREAMING"
}
]
```
NOTE: JSON arguments must include options and values as their own items in the list.
Command (using step.json):
```
aws emr add-steps --cluster-id j-XXXXXXXX --steps file://./step.json
```
Output:
```
{
"StepIds":[
"s-XXXXXXXX",
"s-YYYYYYYY"
]
}
```
**3. To add a Streaming step with multiple files to a cluster (JSON only)**
JSON (multiplefiles.json):
```
[
{
"Name": "JSON Streaming Step",
"Type": "STREAMING",
"ActionOnFailure": "CONTINUE",
"Args": [
"-files",
"s3://amzn-s3-demo-bucket/mapper.py,s3://amzn-s3-demo-bucket/reducer.py",
"-mapper",
"mapper.py",
"-reducer",
"reducer.py",
"-input",
"s3://amzn-s3-demo-bucket/input",
"-output",
"s3://amzn-s3-demo-bucket/output"]
}
]
```
Command:
```
aws emr add-steps --cluster-id j-XXXXXXXX --steps file://./multiplefiles.json
```
Required parameters:
```
Type, Args
```
Optional parameters:
```
Name, ActionOnFailure
```
Output:
```
{
"StepIds":[
"s-XXXXXXXX",
]
}
```
**4. To add Hive steps to a cluster**
Command:
```
aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=HIVE,Name='Hive program',ActionOnFailure=CONTINUE,Args=[-f,s3://amzn-s3-demo-bucket/myhivescript.q,-d,INPUT=s3://amzn-s3-demo-bucket/myhiveinput,-d,OUTPUT=s3://amzn-s3-demo-bucket/myhiveoutput,arg1,arg2] Type=HIVE,Name='Hive steps',ActionOnFailure=TERMINATE_CLUSTER,Args=[-f,s3://elasticmapreduce/samples/hive-ads/libs/model-build.q,-d,INPUT=s3://elasticmapreduce/samples/hive-ads/tables,-d,OUTPUT=s3://amzn-s3-demo-bucket/hive-ads/output/2014-04-18/11-07-32,-d,LIBS=s3://elasticmapreduce/samples/hive-ads/libs]
```
Required parameters:
```
Type, Args
```
Optional parameters:
```
Name, ActionOnFailure
```
Output:
```
{
"StepIds":[
"s-XXXXXXXX",
"s-YYYYYYYY"
]
}
```
**5. To add Pig steps to a cluster**
Command:
```
aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=PIG,Name='Pig program',ActionOnFailure=CONTINUE,Args=[-f,s3://amzn-s3-demo-bucket/mypigscript.pig,-p,INPUT=s3://amzn-s3-demo-bucket/mypiginput,-p,OUTPUT=s3://amzn-s3-demo-bucket/mypigoutput,arg1,arg2] Type=PIG,Name='Pig program',Args=[-f,s3://elasticmapreduce/samples/pig-apache/do-reports2.pig,-p,INPUT=s3://elasticmapreduce/samples/pig-apache/input,-p,OUTPUT=s3://amzn-s3-demo-bucket/pig-apache/output,arg1,arg2]
```
Required parameters:
```
Type, Args
```
Optional parameters:
```
Name, ActionOnFailure
```
Output:
```
{
"StepIds":[
"s-XXXXXXXX",
"s-YYYYYYYY"
]
}
```
**6. To add Impala steps to a cluster**
Command:
```
aws emr add-steps --cluster-id j-XXXXXXXX --steps Type=IMPALA,Name='Impala program',ActionOnFailure=CONTINUE,Args=--impala-script,s3://myimpala/input,--console-output-path,s3://myimpala/output
```
Required parameters:
```
Type, Args
```
Optional parameters:
```
Name, ActionOnFailure
```
Output:
```
{
"StepIds":[
"s-XXXXXXXX",
"s-YYYYYYYY"
]
}
```
+ For API details, see [AddSteps](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/add-steps.html) in *AWS CLI Command Reference*.
### `add-tags`
The following code example shows how to use `add-tags`.
**AWS CLI**
**1. To add tags to a cluster**
Command:
```
aws emr add-tags --resource-id j-xxxxxxx --tags name="John Doe" age=29 sex=male address="123 East NW Seattle"
```
Output:
```
None
```
**2. To list tags of a cluster**
--Command:
```
aws emr describe-cluster --cluster-id j-XXXXXXYY --query Cluster.Tags
```
Output:
```
[
{
"Value": "male",
"Key": "sex"
},
{
"Value": "123 East NW Seattle",
"Key": "address"
},
{
"Value": "John Doe",
"Key": "name"
},
{
"Value": "29",
"Key": "age"
}
]
```
+ For API details, see [AddTags](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/add-tags.html) in *AWS CLI Command Reference*.
### `create-cluster-examples`
The following code example shows how to use `create-cluster-examples`.
**AWS CLI**
Most of the following examples assume that you specified your Amazon EMR service role and Amazon EC2 instance profile. If you have not done this, you must specify each required IAM role or use the `--use-default-roles` parameter when creating your cluster. For more information about specifying IAM roles, see [Configure IAM Roles for Amazon EMR Permissions to AWS Services](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-roles.html) in the *Amazon EMR Management Guide*.
**Example 1: To create a cluster**
The following `create-cluster` example creates a simple EMR cluster.
```
aws emr create-cluster \
--release-label emr-5.14.0 \
--instance-type m4.large \
--instance-count 2
```
This command produces no output.
**Example 2: To create an Amazon EMR cluster with default ServiceRole and InstanceProfile roles**
The following `create-cluster` example creates an Amazon EMR cluster that uses the `--instance-groups` configuration.
```
aws emr create-cluster \
--release-label emr-5.14.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
**Example 3: To create an Amazon EMR cluster that uses an instance fleet**
The following `create-cluster` example creates an Amazon EMR cluster that uses the `--instance-fleets` configuration, specifying two instance types for each fleet and two EC2 Subnets.
```
aws emr create-cluster \
--release-label emr-5.14.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole,SubnetIds=['subnet-ab12345c','subnet-de67890f'] \
--instance-fleets InstanceFleetType=MASTER,TargetOnDemandCapacity=1,InstanceTypeConfigs=['{InstanceType=m4.large}'] InstanceFleetType=CORE,TargetSpotCapacity=11,InstanceTypeConfigs=['{InstanceType=m4.large,BidPrice=0.5,WeightedCapacity=3}','{InstanceType=m4.2xlarge,BidPrice=0.9,WeightedCapacity=5}'],LaunchSpecifications={SpotSpecification='{TimeoutDurationMinutes=120,TimeoutAction=SWITCH_TO_ON_DEMAND}'}
```
**Example 4: To create a cluster with default roles**
The following `create-cluster` example uses the `--use-default-roles` parameter to specify the default service role and instance profile.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--use-default-roles \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 5: To create a cluster and specify the applications to install**
The following `create-cluster` example uses the `--applications` parameter to specify the applications that Amazon EMR installs. This example installs Hadoop, Hive and Pig.
```
aws emr create-cluster \
--applications Name=Hadoop Name=Hive Name=Pig \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 6: To create a cluster that includes Spark**
The following example installs Spark.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--applications Name=Spark \
--ec2-attributes KeyName=myKey \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 7: To specify a custom AMI to use for cluster instances**
The following `create-cluster` example creates a cluster instance based on the Amazon Linux AMI with ID `ami-a518e6df`.
```
aws emr create-cluster \
--name "Cluster with My Custom AMI" \
--custom-ami-id ami-a518e6df \
--ebs-root-volume-size 20 \
--release-label emr-5.9.0 \
--use-default-roles \
--instance-count 2 \
--instance-type m4.large
```
**Example 8: To customize application configurations**
The following examples use the `--configurations` parameter to specify a JSON configuration file that contains application customizations for Hadoop. For more information, see [Configuring Applications](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html) in the *Amazon EMR Release Guide*.
Contents of `configurations.json`:
```
[
{
"Classification": "mapred-site",
"Properties": {
"mapred.tasktracker.map.tasks.maximum": 2
}
},
{
"Classification": "hadoop-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"HADOOP_DATANODE_HEAPSIZE": 2048,
"HADOOP_NAMENODE_OPTS": "-XX:GCTimeRatio=19"
}
}
]
}
]
```
The following example references `configurations.json` as a local file.
```
aws emr create-cluster \
--configurations file://configurations.json \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
The following example references `configurations.json` as a file in Amazon S3.
```
aws emr create-cluster \
--configurations https://s3.amazonaws.com/amzn-s3-demo-bucket/configurations.json \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 9: To create a cluster with master, core, and task instance groups**
The following `create-cluster` example uses `--instance-groups` to specify the type and number of EC2 instances to use for master, core, and task instance groups.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--instance-groups Name=Master,InstanceGroupType=MASTER,InstanceType=m4.large,InstanceCount=1 Name=Core,InstanceGroupType=CORE,InstanceType=m4.large,InstanceCount=2 Name=Task,InstanceGroupType=TASK,InstanceType=m4.large,InstanceCount=2
```
**Example 10: To specify that a cluster should terminate after completing all steps**
The following `create-cluster` example uses `--auto-terminate` to specify that the cluster should shut down automatically after completing all steps.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 11: To specify cluster configuration details such as the Amazon EC2 key pair, network configuration, and security groups**
The following `create-cluster` example creates a cluster with the Amazon EC2 key pair named `myKey` and a customized instance profile named `myProfile`. Key pairs are used to authorize SSH connections to cluster nodes, most often the master node. For more information, see [Use an Amazon EC2 Key Pair for SSH Credentials](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-access-ssh.html) in the *Amazon EMR Management Guide*.
```
aws emr create-cluster \
--ec2-attributes KeyName=myKey,InstanceProfile=myProfile \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
The following example creates a cluster in an Amazon VPC subnet.
```
aws emr create-cluster \
--ec2-attributes SubnetId=subnet-xxxxx \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
The following example creates a cluster in the `us-east-1b` availability zone.
```
aws emr create-cluster \
--ec2-attributes AvailabilityZone=us-east-1b \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
The following example creates a cluster and specifies only the Amazon EMR-managed security groups.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--service-role myServiceRole \
--ec2-attributes InstanceProfile=myRole,EmrManagedMasterSecurityGroup=sg-master1,EmrManagedSlaveSecurityGroup=sg-slave1 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
The following example creates a cluster and specifies only additional Amazon EC2 security groups.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--service-role myServiceRole \
--ec2-attributes InstanceProfile=myRole,AdditionalMasterSecurityGroups=[sg-addMaster1,sg-addMaster2,sg-addMaster3,sg-addMaster4],AdditionalSlaveSecurityGroups=[sg-addSlave1,sg-addSlave2,sg-addSlave3,sg-addSlave4] \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
The following example creates a cluster and specifies the EMR-Managed security groups, as well as additional security groups.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--service-role myServiceRole \
--ec2-attributes InstanceProfile=myRole,EmrManagedMasterSecurityGroup=sg-master1,EmrManagedSlaveSecurityGroup=sg-slave1,AdditionalMasterSecurityGroups=[sg-addMaster1,sg-addMaster2,sg-addMaster3,sg-addMaster4],AdditionalSlaveSecurityGroups=[sg-addSlave1,sg-addSlave2,sg-addSlave3,sg-addSlave4] \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
The following example creates a cluster in a VPC private subnet and use a specific Amazon EC2 security group to enable Amazon EMR service access, which is required for clusters in private subnets.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--service-role myServiceRole \
--ec2-attributes InstanceProfile=myRole,ServiceAccessSecurityGroup=sg-service-access,EmrManagedMasterSecurityGroup=sg-master,EmrManagedSlaveSecurityGroup=sg-slave \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
The following example specifies security group configuration parameters using a JSON file named `ec2_attributes.json` that is stored locally. NOTE: JSON arguments must include options and values as their own items in the list.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--service-role myServiceRole \
--ec2-attributes file://ec2_attributes.json \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
Contents of `ec2_attributes.json`:
```
[
{
"SubnetId": "subnet-xxxxx",
"KeyName": "myKey",
"InstanceProfile":"myRole",
"EmrManagedMasterSecurityGroup": "sg-master1",
"EmrManagedSlaveSecurityGroup": "sg-slave1",
"ServiceAccessSecurityGroup": "sg-service-access",
"AdditionalMasterSecurityGroups": ["sg-addMaster1","sg-addMaster2","sg-addMaster3","sg-addMaster4"],
"AdditionalSlaveSecurityGroups": ["sg-addSlave1","sg-addSlave2","sg-addSlave3","sg-addSlave4"]
}
]
```
**Example 12: To enable debugging and specify a log URI**
The following `create-cluster` example uses the `--enable-debugging` parameter, which allows you to view log files more easily using the debugging tool in the Amazon EMR console. The `--log-uri` parameter is required with `--enable-debugging`.
```
aws emr create-cluster \
--enable-debugging \
--log-uri s3://amzn-s3-demo-bucket/myLog \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 13: To add tags when creating a cluster**
Tags are key-value pairs that help you identify and manage clusters. The following `create-cluster` example uses the `--tags` parameter to create three tags for a cluster, one with the key name `name` and the value `Shirley Rodriguez`, a second with the key name `age` and the value `29`, and a third tag with the key name `department` and the value `Analytics`.
```
aws emr create-cluster \
--tags name="Shirley Rodriguez" age=29 department="Analytics" \
--release-label emr-5.32.0 \
--instance-type m5.xlarge \
--instance-count 3 \
--use-default-roles
```
The following example lists the tags applied to a cluster.
```
aws emr describe-cluster \
--cluster-id j-XXXXXXYY \
--query Cluster.Tags
```
**Example 14: To use a security configuration that enables encryption and other security features**
The following `create-cluster` example uses the `--security-configuration` parameter to specify a security configuration for an EMR cluster. You can use security configurations with Amazon EMR version 4.8.0 or later.
```
aws emr create-cluster \
--instance-type m4.large \
--release-label emr-5.9.0 \
--security-configuration mySecurityConfiguration
```
**Example 15: To create a cluster with additional EBS storage volumes configured for the instance groups**
When specifying additional EBS volumes, the following arguments are required: `VolumeType`, `SizeInGB` if `EbsBlockDeviceConfigs` is specified.
The following `create-cluster` example creates a cluster with multiple EBS volumes attached to EC2 instances in the core instance group.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--use-default-roles \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=d2.xlarge 'InstanceGroupType=CORE,InstanceCount=2,InstanceType=d2.xlarge,EbsConfiguration={EbsOptimized=true,EbsBlockDeviceConfigs=[{VolumeSpecification={VolumeType=gp2,SizeInGB=100}},{VolumeSpecification={VolumeType=io1,SizeInGB=100,Iops=100},VolumesPerInstance=4}]}' \
--auto-terminate
```
The following example creates a cluster with multiple EBS volumes attached to EC2 instances in the master instance group.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--use-default-roles \
--instance-groups 'InstanceGroupType=MASTER, InstanceCount=1, InstanceType=d2.xlarge, EbsConfiguration={EbsOptimized=true, EbsBlockDeviceConfigs=[{VolumeSpecification={VolumeType=io1, SizeInGB=100, Iops=100}},{VolumeSpecification={VolumeType=standard,SizeInGB=50},VolumesPerInstance=3}]}' InstanceGroupType=CORE,InstanceCount=2,InstanceType=d2.xlarge \
--auto-terminate
```
**Example 16: To create a cluster with an automatic scaling policy**
You can attach automatic scaling policies to core and task instance groups using Amazon EMR version 4.0 and later. The automatic scaling policy dynamically adds and removes EC2 instances in response to an Amazon CloudWatch metric. For more information, see Using Automatic Scaling in Amazon EMR `\$1 in the *Amazon EMR Management Guide*.
When attaching an automatic scaling policy, you must also specify the default role for automatic scaling using `--auto-scaling-role EMR_AutoScaling_DefaultRole`.
The following `create-cluster` example specifies the automatic scaling policy for the `CORE` instance group using the `AutoScalingPolicy` argument with an embedded JSON structure, which specifies the scaling policy configuration. Instance groups with an embedded JSON structure must have the entire collection of arguments enclosed in single quotes. Using single quotes is optional for instance groups without an embedded JSON structure.
```
aws emr create-cluster
--release-label emr-5.9.0 \
--use-default-roles --auto-scaling-role EMR_AutoScaling_DefaultRole \
--instance-groups InstanceGroupType=MASTER,InstanceType=d2.xlarge,InstanceCount=1 'InstanceGroupType=CORE,InstanceType=d2.xlarge,InstanceCount=2,AutoScalingPolicy={Constraints={MinCapacity=1,MaxCapacity=5},Rules=[{Name=TestRule,Description=TestDescription,Action={Market=ON_DEMAND,SimpleScalingPolicyConfiguration={AdjustmentType=EXACT_CAPACITY,ScalingAdjustment=2}},Trigger={CloudWatchAlarmDefinition={ComparisonOperator=GREATER_THAN,EvaluationPeriods=5,MetricName=TestMetric,Namespace=EMR,Period=3,Statistic=MAXIMUM,Threshold=4.5,Unit=NONE,Dimensions=[{Key=TestKey,Value=TestValue}]}}}]}'
```
The following example uses a JSON file, `instancegroupconfig.json`, to specify the configuration of all instance groups in a cluster. The JSON file specifies the automatic scaling policy configuration for the core instance group.
```
aws emr create-cluster \
--release-label emr-5.9.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--instance-groups file://myfolder/instancegroupconfig.json \
--auto-scaling-role EMR_AutoScaling_DefaultRole
```
Contents of `instancegroupconfig.json`:
```
[
{
"InstanceCount": 1,
"Name": "MyMasterIG",
"InstanceGroupType": "MASTER",
"InstanceType": "m4.large"
},
{
"InstanceCount": 2,
"Name": "MyCoreIG",
"InstanceGroupType": "CORE",
"InstanceType": "m4.large",
"AutoScalingPolicy": {
"Constraints": {
"MinCapacity": 2,
"MaxCapacity": 10
},
"Rules": [
{
"Name": "Default-scale-out",
"Description": "Replicates the default scale-out rule in the console for YARN memory.",
"Action": {
"SimpleScalingPolicyConfiguration": {
"AdjustmentType": "CHANGE_IN_CAPACITY",
"ScalingAdjustment": 1,
"CoolDown": 300
}
},
"Trigger": {
"CloudWatchAlarmDefinition": {
"ComparisonOperator": "LESS_THAN",
"EvaluationPeriods": 1,
"MetricName": "YARNMemoryAvailablePercentage",
"Namespace": "AWS/ElasticMapReduce",
"Period": 300,
"Threshold": 15,
"Statistic": "AVERAGE",
"Unit": "PERCENT",
"Dimensions": [
{
"Key": "JobFlowId",
"Value": "${emr.clusterId}"
}
]
}
}
}
]
}
}
]
```
**Example 17: Add custom JAR steps when creating a cluster**
The following `create-cluster` example adds steps by specifying a JAR file stored in Amazon S3. Steps submit work to a cluster. The main function defined in the JAR file executes after EC2 instances are provisioned, any bootstrap actions have executed, and applications are installed. The steps are specified using `Type=CUSTOM_JAR`.
Custom JAR steps require the `Jar=` parameter, which specifies the path and file name of the JAR. Optional parameters are `Type`, `Name`, `ActionOnFailure`, `Args`, and `MainClass`. If main class is not specified, the JAR file should specify `Main-Class` in its manifest file.
```
aws emr create-cluster \
--steps Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,Args=arg1,arg2,arg3 Type=CUSTOM_JAR,Name=CustomJAR,ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,MainClass=mymainclass,Args=arg1,arg2,arg3 \
--release-label emr-5.3.1 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 18: To add streaming steps when creating a cluster**
The following `create-cluster` examples add a streaming step to a cluster that terminates after all steps run. Streaming steps require parameters `Type` and `Args`. Streaming steps optional parameters are `Name` and `ActionOnFailure`.
The following example specifies the step inline.
```
aws emr create-cluster \
--steps Type=STREAMING,Name='Streaming Program',ActionOnFailure=CONTINUE,Args=[-files,s3://elasticmapreduce/samples/wordcount/wordSplitter.py,-mapper,wordSplitter.py,-reducer,aggregate,-input,s3://elasticmapreduce/samples/wordcount/input,-output,s3://amzn-s3-demo-bucket/wordcount/output] \
--release-label emr-5.3.1 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
The following example uses a locally stored JSON configuration file named `multiplefiles.json`. The JSON configuration specifies multiple files. To specify multiple files within a step, you must use a JSON configuration file to specify the step. JSON arguments must include options and values as their own items in the list.
```
aws emr create-cluster \
--steps file://./multiplefiles.json \
--release-label emr-5.9.0 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
Contents of `multiplefiles.json`:
```
[
{
"Name": "JSON Streaming Step",
"Args": [
"-files",
"s3://elasticmapreduce/samples/wordcount/wordSplitter.py",
"-mapper",
"wordSplitter.py",
"-reducer",
"aggregate",
"-input",
"s3://elasticmapreduce/samples/wordcount/input",
"-output",
"s3://amzn-s3-demo-bucket/wordcount/output"
],
"ActionOnFailure": "CONTINUE",
"Type": "STREAMING"
}
]
```
**Example 19: To add Hive steps when creating a cluster**
The following example add Hive steps when creating a cluster. Hive steps require parameters `Type` and `Args`. Hive steps optional parameters are `Name` and `ActionOnFailure`.
```
aws emr create-cluster \
--steps Type=HIVE,Name='Hive program',ActionOnFailure=CONTINUE,ActionOnFailure=TERMINATE_CLUSTER,Args=[-f,s3://elasticmapreduce/samples/hive-ads/libs/model-build.q,-d,INPUT=s3://elasticmapreduce/samples/hive-ads/tables,-d,OUTPUT=s3://amzn-s3-demo-bucket/hive-ads/output/2014-04-18/11-07-32,-d,LIBS=s3://elasticmapreduce/samples/hive-ads/libs] \
--applications Name=Hive \
--release-label emr-5.3.1 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
**Example 20: To add Pig steps when creating a cluster**
The following example adds Pig steps when creating a cluster. Pig steps required parameters are `Type` and `Args`. Pig steps optional parameters are `Name` and `ActionOnFailure`.
```
aws emr create-cluster \
--steps Type=PIG,Name='Pig program',ActionOnFailure=CONTINUE,Args=[-f,s3://elasticmapreduce/samples/pig-apache/do-reports2.pig,-p,INPUT=s3://elasticmapreduce/samples/pig-apache/input,-p,OUTPUT=s3://amzn-s3-demo-bucket/pig-apache/output] \
--applications Name=Pig \
--release-label emr-5.3.1 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
**Example 21: To add bootstrap actions**
The following `create-cluster` example runs two bootstrap actions defined as scripts that are stored in Amazon S3.
```
aws emr create-cluster \
--bootstrap-actions Path=s3://amzn-s3-demo-bucket/myscript1,Name=BootstrapAction1,Args=[arg1,arg2] Path=s3://amzn-s3-demo-bucket/myscript2,Name=BootstrapAction2,Args=[arg1,arg2] \
--release-label emr-5.3.1 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--auto-terminate
```
**Example 22: To enable EMRFS consistent view and customize the RetryCount and RetryPeriod settings**
The following `create-cluster` example specifies the retry count and retry period for EMRFS consistent view. The `Consistent=true` argument is required.
```
aws emr create-cluster \
--instance-type m4.large \
--release-label emr-5.9.0 \
--emrfs Consistent=true,RetryCount=6,RetryPeriod=30
```
The following example specifies the same EMRFS configuration as the previous example, using a locally stored JSON configuration file named `emrfsconfig.json`.
```
aws emr create-cluster \
--instance-type m4.large \
--release-label emr-5.9.0 \
--emrfs file://emrfsconfig.json
```
Contents of `emrfsconfig.json`:
```
{
"Consistent": true,
"RetryCount": 6,
"RetryPeriod": 30
}
```
**Example 23: To create a cluster with Kerberos configured**
The following `create-cluster` examples create a cluster using a security configuration with Kerberos enabled, and establishes Kerberos parameters for the cluster using `--kerberos-attributes`.
The following command specifies Kerberos attributes for the cluster inline.
```
aws emr create-cluster \
--instance-type m3.xlarge \
--release-label emr-5.10.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--security-configuration mySecurityConfiguration \
--kerberos-attributes Realm=EC2.INTERNAL,KdcAdminPassword=123,CrossRealmTrustPrincipalPassword=123
```
The following command specifies the same attributes, but references a locally stored JSON file named `kerberos_attributes.json`. In this example, the file is saved in the same directory where you run the command. You can also reference a configuration file saved in Amazon S3.
```
aws emr create-cluster \
--instance-type m3.xlarge \
--release-label emr-5.10.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--security-configuration mySecurityConfiguration \
--kerberos-attributes file://kerberos_attributes.json
```
Contents of `kerberos_attributes.json`:
```
{
"Realm": "EC2.INTERNAL",
"KdcAdminPassword": "123",
"CrossRealmTrustPrincipalPassword": "123",
}
```
The following `create-cluster` example creates an Amazon EMR cluster that uses the `--instance-groups` configuration and has a managed scaling policy.
```
aws emr create-cluster \
--release-label emr-5.30.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
--managed-scaling-policy ComputeLimits='{MinimumCapacityUnits=2,MaximumCapacityUnits=4,UnitType=Instances}'
```
The following `create-cluster` example creates an Amazon EMR cluster that uses the "--log-encryption-kms-key-id" to define KMS key ID utilized for Log encryption.
```
aws emr create-cluster \
--release-label emr-5.30.0 \
--log-uri s3://amzn-s3-demo-bucket/myLog \
--log-encryption-kms-key-id arn:aws:kms:us-east-1:110302272565:key/dd559181-283e-45d7-99d1-66da348c4d33 \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large
```
The following `create-cluster` example creates an Amazon EMR cluster that uses the "--placement-group-configs" configuration to place master nodes in a high-availability (HA) cluster within an EC2 placement group using `SPREAD` placement strategy.
```
aws emr create-cluster \
--release-label emr-5.30.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--instance-groups InstanceGroupType=MASTER,InstanceCount=3,InstanceType=m4.largeInstanceGroupType=CORE,InstanceCount=1,InstanceType=m4.large \
--placement-group-configs InstanceRole=MASTER
```
The following `create-cluster` example creates an Amazon EMR cluster that uses the "--auto-termination-policy" configuration to place an automatic idle termination threshold for the cluster.
```
aws emr create-cluster \
--release-label emr-5.34.0 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=1,InstanceType=m4.large \
--auto-termination-policy IdleTimeout=100
```
The following `create-cluster` example creates an Amazon EMR cluster that uses the "--os-release-label" to define an Amazon Linux release for cluster launch
```
aws emr create-cluster \
--release-label emr-6.6.0 \
--os-release-label 2.0.20220406.1 \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large InstanceGroupType=CORE,InstanceCount=1,InstanceType=m4.large
```
**Example 24: To specify an EBS root volume attributes: size, iops and throughput for cluster instances created with EMR releases 6.15.0 and later**
The following `create-cluster` example creates an Amazon EMR cluster that uses root volume attributes to configure root volumes specifications for the EC2 instances.
```
aws emr create-cluster \
--name "Cluster with My Custom AMI" \
--custom-ami-id ami-a518e6df \
--ebs-root-volume-size 20 \
--ebs-root-volume-iops 3000 \
--ebs-root-volume-throughput 125 \
--release-label emr-6.15.0 \
--use-default-roles \
--instance-count 2 \
--instance-type m4.large
```
+ For API details, see [CreateClusterExamples](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/create-cluster-examples.html) in *AWS CLI Command Reference*.
### `create-default-roles`
The following code example shows how to use `create-default-roles`.
**AWS CLI**
**1. To create the default IAM role for EC2**
Command:
```
aws emr create-default-roles
```
Output:
```
If the role already exists then the command returns nothing.
If the role does not exist then the output will be:
[
{
"RolePolicy": {
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"cloudwatch:*",
"dynamodb:*",
"ec2:Describe*",
"elasticmapreduce:Describe*",
"elasticmapreduce:ListBootstrapActions",
"elasticmapreduce:ListClusters",
"elasticmapreduce:ListInstanceGroups",
"elasticmapreduce:ListInstances",
"elasticmapreduce:ListSteps",
"kinesis:CreateStream",
"kinesis:DeleteStream",
"kinesis:DescribeStream",
"kinesis:GetRecords",
"kinesis:GetShardIterator",
"kinesis:MergeShards",
"kinesis:PutRecord",
"kinesis:SplitShard",
"rds:Describe*",
"s3:*",
"sdb:*",
"sns:*",
"sqs:*"
],
"Resource": "*",
"Effect": "Allow"
}
]
},
"Role": {
"AssumeRolePolicyDocument": {
"Version":"2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
}
}
]
},
"RoleId": "AROAIQ5SIQUGL5KMYBJX6",
"CreateDate": "2015-06-09T17:09:04.602Z",
"RoleName": "EMR_EC2_DefaultRole",
"Path": "/",
"Arn": "arn:aws:iam::176430881729:role/EMR_EC2_DefaultRole"
}
},
{
"RolePolicy": {
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CancelSpotInstanceRequests",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeAccountAttributes",
"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeKeyPairs",
"ec2:DescribePrefixLists",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSpotInstanceRequests",
"ec2:DescribeSpotPriceHistory",
"ec2:DescribeSubnets",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeVpcEndpointServices",
"ec2:DescribeVpcs",
"ec2:ModifyImageAttribute",
"ec2:ModifyInstanceAttribute",
"ec2:RequestSpotInstances",
"ec2:RunInstances",
"ec2:TerminateInstances",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:ListInstanceProfiles",
"iam:ListRolePolicies",
"iam:PassRole",
"s3:CreateBucket",
"s3:Get*",
"s3:List*",
"sdb:BatchPutAttributes",
"sdb:Select",
"sqs:CreateQueue",
"sqs:Delete*",
"sqs:GetQueue*",
"sqs:ReceiveMessage"
],
"Resource": "*",
"Effect": "Allow"
}
]
},
"Role": {
"AssumeRolePolicyDocument": {
"Version":"2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "elasticmapreduce.amazonaws.com"
}
}
]
},
"RoleId": "AROAI3SRVPPVSRDLARBPY",
"CreateDate": "2015-06-09T17:09:10.401Z",
"RoleName": "EMR_DefaultRole",
"Path": "/",
"Arn": "arn:aws:iam::176430881729:role/EMR_DefaultRole"
}
}
]
```
+ For API details, see [CreateDefaultRoles](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/create-default-roles.html) in *AWS CLI Command Reference*.
### `create-security-configuration`
The following code example shows how to use `create-security-configuration`.
**AWS CLI**
**1. To create a security configuration with in-transit encryption enabled with PEM for certificate provider, and at-rest encryption enabled with SSE-S3 for S3 encryption and AWS-KMS for local disk key provider**
Command:
```
aws emr create-security-configuration --name MySecurityConfig --security-configuration '{
"EncryptionConfiguration": {
"EnableInTransitEncryption" : true,
"EnableAtRestEncryption" : true,
"InTransitEncryptionConfiguration" : {
"TLSCertificateConfiguration" : {
"CertificateProviderType" : "PEM",
"S3Object" : "s3://mycertstore/artifacts/MyCerts.zip"
}
},
"AtRestEncryptionConfiguration" : {
"S3EncryptionConfiguration" : {
"EncryptionMode" : "SSE-S3"
},
"LocalDiskEncryptionConfiguration" : {
"EncryptionKeyProviderType" : "AwsKms",
"AwsKmsKey" : "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
}
}
}
}'
```
Output:
```
{
"CreationDateTime": 1474070889.129,
"Name": "MySecurityConfig"
}
```
JSON equivalent (contents of security\$1configuration.json):
```
{
"EncryptionConfiguration": {
"EnableInTransitEncryption": true,
"EnableAtRestEncryption": true,
"InTransitEncryptionConfiguration": {
"TLSCertificateConfiguration": {
"CertificateProviderType": "PEM",
"S3Object": "s3://mycertstore/artifacts/MyCerts.zip"
}
},
"AtRestEncryptionConfiguration": {
"S3EncryptionConfiguration": {
"EncryptionMode": "SSE-S3"
},
"LocalDiskEncryptionConfiguration": {
"EncryptionKeyProviderType": "AwsKms",
"AwsKmsKey": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
}
}
}
}
```
Command (using security\$1configuration.json):
```
aws emr create-security-configuration --name "MySecurityConfig" --security-configuration file://./security_configuration.json
```
Output:
```
{
"CreationDateTime": 1474070889.129,
"Name": "MySecurityConfig"
}
```
**2. To create a security configuration with Kerberos enabled using cluster-dedicated KDC and cross-realm trust**
Command:
```
aws emr create-security-configuration --name MySecurityConfig --security-configuration '{
"AuthenticationConfiguration": {
"KerberosConfiguration": {
"Provider": "ClusterDedicatedKdc",
"ClusterDedicatedKdcConfiguration": {
"TicketLifetimeInHours": 24,
"CrossRealmTrustConfiguration": {
"Realm": "AD.DOMAIN.COM",
"Domain": "ad.domain.com",
"AdminServer": "ad.domain.com",
"KdcServer": "ad.domain.com"
}
}
}
}
}'
```
Output:
```
{
"CreationDateTime": 1490225558.982,
"Name": "MySecurityConfig"
}
```
JSON equivalent (contents of security\$1configuration.json):
```
{
"AuthenticationConfiguration": {
"KerberosConfiguration": {
"Provider": "ClusterDedicatedKdc",
"ClusterDedicatedKdcConfiguration": {
"TicketLifetimeInHours": 24,
"CrossRealmTrustConfiguration": {
"Realm": "AD.DOMAIN.COM",
"Domain": "ad.domain.com",
"AdminServer": "ad.domain.com",
"KdcServer": "ad.domain.com"
}
}
}
}
}
```
Command (using security\$1configuration.json):
```
aws emr create-security-configuration --name "MySecurityConfig" --security-configuration file://./security_configuration.json
```
Output:
```
{
"CreationDateTime": 1490225558.982,
"Name": "MySecurityConfig"
}
```
+ For API details, see [CreateSecurityConfiguration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/create-security-configuration.html) in *AWS CLI Command Reference*.
### `delete-security-configuration`
The following code example shows how to use `delete-security-configuration`.
**AWS CLI**
**To delete a security configuration in the current region**
Command:
```
aws emr delete-security-configuration --name MySecurityConfig
```
Output:
```
None
```
+ For API details, see [DeleteSecurityConfiguration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/delete-security-configuration.html) in *AWS CLI Command Reference*.
### `describe-cluster`
The following code example shows how to use `describe-cluster`.
**AWS CLI**
Command:
```
aws emr describe-cluster --cluster-id j-XXXXXXXX
```
Output:
```
For release-label based uniform instance groups cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1436475075.199,
"CreationDateTime": 1436474656.563,
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"ServiceAccessSecurityGroup": "sg-xxxxxxxx",
"EmrManagedMasterSecurityGroup": "sg-xxxxxxxx",
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2KeyName": "myKey",
"Ec2AvailabilityZone": "us-east-1c",
"EmrManagedSlaveSecurityGroup": "sg-yyyyyyyyy"
},
"Name": "My Cluster",
"ServiceRole": "EMR_DefaultRole",
"Tags": [],
"TerminationProtected": true,
"UnhealthyNodeReplacement": true,
"ReleaseLabel": "emr-4.0.0",
"NormalizedInstanceHours": 96,
"InstanceGroups": [
{
"RequestedInstanceCount": 2,
"Status": {
"Timeline": {
"ReadyDateTime": 1436475074.245,
"CreationDateTime": 1436474656.564,
"EndDateTime": 1436638158.387
},
"State": "RUNNING",
"StateChangeReason": {
"Message": "",
}
},
"Name": "CORE",
"InstanceGroupType": "CORE",
"Id": "ig-YYYYYYY",
"Configurations": [],
"InstanceType": "m3.large",
"Market": "ON_DEMAND",
"RunningInstanceCount": 2
},
{
"RequestedInstanceCount": 1,
"Status": {
"Timeline": {
"ReadyDateTime": 1436475074.245,
"CreationDateTime": 1436474656.564,
"EndDateTime": 1436638158.387
},
"State": "RUNNING",
"StateChangeReason": {
"Message": "",
}
},
"Name": "MASTER",
"InstanceGroupType": "MASTER",
"Id": "ig-XXXXXXXXX",
"Configurations": [],
"InstanceType": "m3.large",
"Market": "ON_DEMAND",
"RunningInstanceCount": 1
}
],
"Applications": [
{
"Name": "Hadoop"
}
],
"VisibleToAllUsers": true,
"BootstrapActions": [],
"MasterPublicDnsName": "ec2-54-147-144-78.compute-1.amazonaws.com",
"AutoTerminate": false,
"Id": "j-XXXXXXXX",
"Configurations": [
{
"Properties": {
"fs.s3.consistent.retryPeriodSeconds": "20",
"fs.s3.enableServerSideEncryption": "true",
"fs.s3.consistent": "false",
"fs.s3.consistent.retryCount": "2"
},
"Classification": "emrfs-site"
}
]
}
}
For release-label based instance fleet cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1487897289.705,
"CreationDateTime": 1487896933.942
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"EmrManagedMasterSecurityGroup": "sg-xxxxx",
"RequestedEc2AvailabilityZones": [],
"RequestedEc2SubnetIds": [],
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2AvailabilityZone": "us-east-1a",
"EmrManagedSlaveSecurityGroup": "sg-xxxxx"
},
"Name": "My Cluster",
"ServiceRole": "EMR_DefaultRole",
"Tags": [],
"TerminationProtected": false,
"UnhealthyNodeReplacement": false,
"ReleaseLabel": "emr-5.2.0",
"NormalizedInstanceHours": 472,
"InstanceCollectionType": "INSTANCE_FLEET",
"InstanceFleets": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1487897212.74,
"CreationDateTime": 1487896933.948
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"ProvisionedSpotCapacity": 1,
"Name": "MASTER",
"InstanceFleetType": "MASTER",
"LaunchSpecifications": {
"SpotSpecification": {
"TimeoutDurationMinutes": 60,
"TimeoutAction": "TERMINATE_CLUSTER"
}
},
"TargetSpotCapacity": 1,
"ProvisionedOnDemandCapacity": 0,
"InstanceTypeSpecifications": [
{
"BidPrice": "0.5",
"InstanceType": "m3.xlarge",
"WeightedCapacity": 1
}
],
"Id": "if-xxxxxxx",
"TargetOnDemandCapacity": 0
}
],
"Applications": [
{
"Version": "2.7.3",
"Name": "Hadoop"
}
],
"ScaleDownBehavior": "TERMINATE_AT_INSTANCE_HOUR",
"VisibleToAllUsers": true,
"BootstrapActions": [],
"MasterPublicDnsName": "ec2-xxx-xx-xxx-xx.compute-1.amazonaws.com",
"AutoTerminate": false,
"Id": "j-xxxxx",
"Configurations": []
}
}
For ami based uniform instance group cluster:
{
"Cluster": {
"Status": {
"Timeline": {
"ReadyDateTime": 1399400564.432,
"CreationDateTime": 1399400268.62
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting for steps to run"
}
},
"Ec2InstanceAttributes": {
"IamInstanceProfile": "EMR_EC2_DefaultRole",
"Ec2AvailabilityZone": "us-east-1c"
},
"Name": "My Cluster",
"Tags": [],
"TerminationProtected": true,
"UnhealthyNodeReplacement": true,
"RunningAmiVersion": "2.5.4",
"InstanceGroups": [
{
"RequestedInstanceCount": 1,
"Status": {
"Timeline": {
"ReadyDateTime": 1399400558.848,
"CreationDateTime": 1399400268.621
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"Name": "Master instance group",
"InstanceGroupType": "MASTER",
"InstanceType": "m1.small",
"Id": "ig-ABCD",
"Market": "ON_DEMAND",
"RunningInstanceCount": 1
},
{
"RequestedInstanceCount": 2,
"Status": {
"Timeline": {
"ReadyDateTime": 1399400564.439,
"CreationDateTime": 1399400268.621
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"Name": "Core instance group",
"InstanceGroupType": "CORE",
"InstanceType": "m1.small",
"Id": "ig-DEF",
"Market": "ON_DEMAND",
"RunningInstanceCount": 2
}
],
"Applications": [
{
"Version": "1.0.3",
"Name": "hadoop"
}
],
"BootstrapActions": [],
"VisibleToAllUsers": false,
"RequestedAmiVersion": "2.4.2",
"LogUri": "s3://myLogUri/",
"AutoTerminate": false,
"Id": "j-XXXXXXXX"
}
}
```
+ For API details, see [DescribeCluster](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/describe-cluster.html) in *AWS CLI Command Reference*.
### `describe-step`
The following code example shows how to use `describe-step`.
**AWS CLI**
The following command describes a step with the step ID `s-3LZC0QUT43AM` in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr describe-step --cluster-id j-3SD91U2E1L2QX --step-id s-3LZC0QUT43AM
```
Output:
```
{
"Step": {
"Status": {
"Timeline": {
"EndDateTime": 1433200470.481,
"CreationDateTime": 1433199926.597,
"StartDateTime": 1433200404.959
},
"State": "COMPLETED",
"StateChangeReason": {}
},
"Config": {
"Args": [
"s3://us-west-2.elasticmapreduce/libs/hive/hive-script",
"--base-path",
"s3://us-west-2.elasticmapreduce/libs/hive/",
"--install-hive",
"--hive-versions",
"0.13.1"
],
"Jar": "s3://us-west-2.elasticmapreduce/libs/script-runner/script-runner.jar",
"Properties": {}
},
"Id": "s-3LZC0QUT43AM",
"ActionOnFailure": "TERMINATE_CLUSTER",
"Name": "Setup hive"
}
}
```
+ For API details, see [DescribeStep](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/describe-step.html) in *AWS CLI Command Reference*.
### `get`
The following code example shows how to use `get`.
**AWS CLI**
The following downloads the `hadoop-examples.jar` archive from the master instance in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr get --cluster-id j-3SD91U2E1L2QX --key-pair-file ~/.ssh/mykey.pem --src /home/hadoop-examples.jar --dest ~
```
+ For API details, see [Get](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/get.html) in *AWS CLI Command Reference*.
### `list-clusters`
The following code example shows how to use `list-clusters`.
**AWS CLI**
The following command lists all active EMR clusters in the current region:
```
aws emr list-clusters --active
```
Output:
```
{
"Clusters": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1433200405.353,
"CreationDateTime": 1433199926.596
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting after step completed"
}
},
"NormalizedInstanceHours": 6,
"Id": "j-3SD91U2E1L2QX",
"Name": "my-cluster"
}
]
}
```
+ For API details, see [ListClusters](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-clusters.html) in *AWS CLI Command Reference*.
### `list-instance-fleets`
The following code example shows how to use `list-instance-fleets`.
**AWS CLI**
**To get configuration details of instance fleets in a cluster**
This example lists the details of instance fleets in the cluster specified.
Command:
```
list-instance-fleets --cluster-id 'j-12ABCDEFGHI34JK'
```
Output:
```
{
"InstanceFleets": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1488759094.637,
"CreationDateTime": 1488758719.817
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"ProvisionedSpotCapacity": 6,
"Name": "CORE",
"InstanceFleetType": "CORE",
"LaunchSpecifications": {
"SpotSpecification": {
"TimeoutDurationMinutes": 60,
"TimeoutAction": "TERMINATE_CLUSTER"
}
},
"ProvisionedOnDemandCapacity": 2,
"InstanceTypeSpecifications": [
{
"BidPrice": "0.5",
"InstanceType": "m3.xlarge",
"WeightedCapacity": 2
}
],
"Id": "if-1ABC2DEFGHIJ3"
},
{
"Status": {
"Timeline": {
"ReadyDateTime": 1488759058.598,
"CreationDateTime": 1488758719.811
},
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
}
},
"ProvisionedSpotCapacity": 0,
"Name": "MASTER",
"InstanceFleetType": "MASTER",
"ProvisionedOnDemandCapacity": 1,
"InstanceTypeSpecifications": [
{
"BidPriceAsPercentageOfOnDemandPrice": 100.0,
"InstanceType": "m3.xlarge",
"WeightedCapacity": 1
}
],
"Id": "if-2ABC4DEFGHIJ4"
}
]
}
```
+ For API details, see [ListInstanceFleets](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-instance-fleets.html) in *AWS CLI Command Reference*.
### `list-instances`
The following code example shows how to use `list-instances`.
**AWS CLI**
The following command lists all of the instances in a cluster with the cluster ID `j-3C6XNQ39VR9WL`:
```
aws emr list-instances --cluster-id j-3C6XNQ39VR9WL
```
Output:
```
For a uniform instance group based cluster
{
"Instances": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1433200400.03,
"CreationDateTime": 1433199960.152
},
"State": "RUNNING",
"StateChangeReason": {}
},
"Ec2InstanceId": "i-f19ecfee",
"PublicDnsName": "ec2-52-52-41-150.us-west-2.compute.amazonaws.com",
"PrivateDnsName": "ip-172-21-11-216.us-west-2.compute.internal",
"PublicIpAddress": "52.52.41.150",
"Id": "ci-3NNHQUQ2TWB6Y",
"PrivateIpAddress": "172.21.11.216"
},
{
"Status": {
"Timeline": {
"ReadyDateTime": 1433200400.031,
"CreationDateTime": 1433199949.102
},
"State": "RUNNING",
"StateChangeReason": {}
},
"Ec2InstanceId": "i-1feee4c2",
"PublicDnsName": "ec2-52-63-246-32.us-west-2.compute.amazonaws.com",
"PrivateDnsName": "ip-172-31-24-130.us-west-2.compute.internal",
"PublicIpAddress": "52.63.246.32",
"Id": "ci-GAOCMKNKDCV7",
"PrivateIpAddress": "172.21.11.215"
},
{
"Status": {
"Timeline": {
"ReadyDateTime": 1433200400.031,
"CreationDateTime": 1433199949.102
},
"State": "RUNNING",
"StateChangeReason": {}
},
"Ec2InstanceId": "i-15cfeee3",
"PublicDnsName": "ec2-52-25-246-63.us-west-2.compute.amazonaws.com",
"PrivateDnsName": "ip-172-31-24-129.us-west-2.compute.internal",
"PublicIpAddress": "52.25.246.63",
"Id": "ci-2W3TDFFB47UAD",
"PrivateIpAddress": "172.21.11.214"
}
]
}
For a fleet based cluster:
{
"Instances": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1487810810.878,
"CreationDateTime": 1487810588.367,
"EndDateTime": 1488022990.924
},
"State": "TERMINATED",
"StateChangeReason": {
"Message": "Instance was terminated."
}
},
"Ec2InstanceId": "i-xxxxx",
"InstanceFleetId": "if-xxxxx",
"EbsVolumes": [],
"PublicDnsName": "ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com",
"InstanceType": "m3.xlarge",
"PrivateDnsName": "ip-xx-xx-xxx-xx.ec2.internal",
"Market": "SPOT",
"PublicIpAddress": "xx.xx.xxx.xxx",
"Id": "ci-xxxxx",
"PrivateIpAddress": "10.47.191.80"
}
]
}
```
+ For API details, see [ListInstances](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-instances.html) in *AWS CLI Command Reference*.
### `list-security-configurations`
The following code example shows how to use `list-security-configurations`.
**AWS CLI**
**To list security configurations in the current region**
Command:
```
aws emr list-security-configurations
```
Output:
```
{
"SecurityConfigurations": [
{
"CreationDateTime": 1473889697.417,
"Name": "MySecurityConfig-1"
},
{
"CreationDateTime": 1473889697.417,
"Name": "MySecurityConfig-2"
}
]
}
```
+ For API details, see [ListSecurityConfigurations](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-security-configurations.html) in *AWS CLI Command Reference*.
### `list-steps`
The following code example shows how to use `list-steps`.
**AWS CLI**
The following command lists all of the steps in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr list-steps --cluster-id j-3SD91U2E1L2QX
```
+ For API details, see [ListSteps](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-steps.html) in *AWS CLI Command Reference*.
### `list-studios`
The following code example shows how to use `list-studios`.
**AWS CLI**
**To list available EMR Studios**
The following `list-studios` example lists the EMR Studios in the AWS account.:
```
aws emr list-studios
```
Output:
```
{
"Studios": [
{
"StudioId": "es-XXXXXXX132E0X7R0W7GAS1MVB",
"Name": "My_EMR_Studio",
"Url": "https://es-XXXXXXX132E0X7R0W7GAS1MVB.emrstudio-prod.us-east-1.amazonaws.com",
"AuthMode": "IAM",
"CreationTime": 1761664173.624
}
]
}
```
For more information, see [Monitor, update and delete Amazon EMR Studio resources](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-manage-studio.html) in the *Amazon EMR Management Guide*.
+ For API details, see [ListStudios](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/list-studios.html) in *AWS CLI Command Reference*.
### `modify-cluster-attributes`
The following code example shows how to use `modify-cluster-attributes`.
**AWS CLI**
The following command sets the visibility of an EMR cluster with the ID `j-301CDNY0J5XM4` to all users:
```
aws emr modify-cluster-attributes --cluster-id j-301CDNY0J5XM4 --visible-to-all-users
```
+ For API details, see [ModifyClusterAttributes](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/modify-cluster-attributes.html) in *AWS CLI Command Reference*.
### `modify-instance-fleet`
The following code example shows how to use `modify-instance-fleet`.
**AWS CLI**
**To change the target capacites of an instance fleet**
This example changes the On-Demand and Spot target capacities to 1 for the instance fleet specified.
Command:
```
aws emr modify-instance-fleet --cluster-id 'j-12ABCDEFGHI34JK' --instance-fleet InstanceFleetId='if-2ABC4DEFGHIJ4',TargetOnDemandCapacity=1,TargetSpotCapacity=1
```
+ For API details, see [ModifyInstanceFleet](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/modify-instance-fleet.html) in *AWS CLI Command Reference*.
### `put`
The following code example shows how to use `put`.
**AWS CLI**
The following command uploads a file named `healthcheck.sh` to the master instance in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr put --cluster-id j-3SD91U2E1L2QX --key-pair-file ~/.ssh/mykey.pem --src ~/scripts/healthcheck.sh --dest /home/hadoop/bin/healthcheck.sh
```
+ For API details, see [Put](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/put.html) in *AWS CLI Command Reference*.
### `remove-tags`
The following code example shows how to use `remove-tags`.
**AWS CLI**
The following command removes a tag with the key `prod` from a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr remove-tags --resource-id j-3SD91U2E1L2QX --tag-keys prod
```
+ For API details, see [RemoveTags](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/remove-tags.html) in *AWS CLI Command Reference*.
### `schedule-hbase-backup`
The following code example shows how to use `schedule-hbase-backup`.
**AWS CLI**
**Note: This command can only be used with HBase on AMI version 2.x and 3.x**
**1. To schedule a full HBase backup** >>>>>>> 06ab6d6e13564b5733d75abaf3b599f93cf39a23
Command:
```
aws emr schedule-hbase-backup --cluster-id j-XXXXXXYY --type full --dir
s3://amzn-s3-demo-bucket/backup --interval 10 --unit hours --start-time
2014-04-21T05:26:10Z --consistent
```
Output:
```
None
```
**2. To schedule an incremental HBase backup**
Command:
```
aws emr schedule-hbase-backup --cluster-id j-XXXXXXYY --type incremental
--dir s3://amzn-s3-demo-bucket/backup --interval 30 --unit minutes --start-time
2014-04-21T05:26:10Z --consistent
```
Output:
```
None
```
+ For API details, see [ScheduleHbaseBackup](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/schedule-hbase-backup.html) in *AWS CLI Command Reference*.
### `socks`
The following code example shows how to use `socks`.
**AWS CLI**
The following command opens a socks connection with the master instance in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr socks --cluster-id j-3SD91U2E1L2QX --key-pair-file ~/.ssh/mykey.pem
```
The key pair file option takes a local path to a private key file.
+ For API details, see [Socks](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/socks.html) in *AWS CLI Command Reference*.
### `ssh`
The following code example shows how to use `ssh`.
**AWS CLI**
The following command opens an ssh connection with the master instance in a cluster with the cluster ID `j-3SD91U2E1L2QX`:
```
aws emr ssh --cluster-id j-3SD91U2E1L2QX --key-pair-file ~/.ssh/mykey.pem
```
The key pair file option takes a local path to a private key file.
Output:
```
ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i /home/local/user/.ssh/mykey.pem hadoop@ec2-52-52-41-150.us-west-2.compute.amazonaws.com
Warning: Permanently added 'ec2-52-52-41-150.us-west-2.compute.amazonaws.com,52.52.41.150' (ECDSA) to the list of known hosts.
Last login: Mon Jun 1 23:15:38 2015
__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/
26 package(s) needed for security, out of 39 available
Run "sudo yum update" to apply all updates.
--------------------------------------------------------------------------------
Welcome to Amazon Elastic MapReduce running Hadoop and Amazon Linux.
Hadoop is installed in /home/hadoop. Log files are in /mnt/var/log/hadoop. Check
/mnt/var/log/hadoop/steps for diagnosing step failures.
The Hadoop UI can be accessed via the following commands:
ResourceManager lynx http://ip-172-21-11-216:9026/
NameNode lynx http://ip-172-21-11-216:9101/
--------------------------------------------------------------------------------
[hadoop@ip-172-31-16-216 ~]$
```
+ For API details, see [Ssh](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/ssh.html) in *AWS CLI Command Reference*.