The AWS::EMR::Cluster
resource specifies an Amazon EMR cluster. This cluster is a collection of Amazon EC2 instances that run open source big data frameworks and applications to process and analyze vast amounts of data. For more information, see the
Amazon EMR Management Guide.
Amazon EMR now supports launching task instance groups and task instance
fleets as part of the AWS::EMR::Cluster
resource. This can be done by using
the JobFlowInstancesConfig
property type's TaskInstanceGroups
and
TaskInstanceFleets
subproperties. Using these subproperties reduces delays
in provisioning task nodes compared to specifying task nodes with the
AWS::EMR::InstanceGroupConfig
and
AWS::EMR::InstanceFleetConfig
resources. Please refer to the examples at
the bottom of this page to learn how to use these subproperties.
Syntax
To declare this entity in your AWS CloudFormation template, use the following syntax:
JSON
{
"Type" : "AWS::EMR::Cluster",
"Properties" : {
"AdditionalInfo" : Json
,
"Applications" : [ Application, ... ]
,
"AutoScalingRole" : String
,
"AutoTerminationPolicy" : AutoTerminationPolicy
,
"BootstrapActions" : [ BootstrapActionConfig, ... ]
,
"Configurations" : [ Configuration, ... ]
,
"CustomAmiId" : String
,
"EbsRootVolumeIops" : Integer
,
"EbsRootVolumeSize" : Integer
,
"EbsRootVolumeThroughput" : Integer
,
"Instances" : JobFlowInstancesConfig
,
"JobFlowRole" : String
,
"KerberosAttributes" : KerberosAttributes
,
"LogEncryptionKmsKeyId" : String
,
"LogUri" : String
,
"ManagedScalingPolicy" : ManagedScalingPolicy
,
"Name" : String
,
"OSReleaseLabel" : String
,
"PlacementGroupConfigs" : [ PlacementGroupConfig, ... ]
,
"ReleaseLabel" : String
,
"ScaleDownBehavior" : String
,
"SecurityConfiguration" : String
,
"ServiceRole" : String
,
"StepConcurrencyLevel" : Integer
,
"Steps" : [ StepConfig, ... ]
,
"Tags" : [ Tag, ... ]
,
"VisibleToAllUsers" : Boolean
}
}
YAML
Type: AWS::EMR::Cluster
Properties:
AdditionalInfo: Json
Applications:
- Application
AutoScalingRole: String
AutoTerminationPolicy:
AutoTerminationPolicy
BootstrapActions:
- BootstrapActionConfig
Configurations:
- Configuration
CustomAmiId: String
EbsRootVolumeIops: Integer
EbsRootVolumeSize: Integer
EbsRootVolumeThroughput: Integer
Instances:
JobFlowInstancesConfig
JobFlowRole: String
KerberosAttributes:
KerberosAttributes
LogEncryptionKmsKeyId: String
LogUri: String
ManagedScalingPolicy:
ManagedScalingPolicy
Name: String
OSReleaseLabel: String
PlacementGroupConfigs:
- PlacementGroupConfig
ReleaseLabel: String
ScaleDownBehavior: String
SecurityConfiguration: String
ServiceRole: String
StepConcurrencyLevel: Integer
Steps:
- StepConfig
Tags:
- Tag
VisibleToAllUsers: Boolean
Properties
AdditionalInfo
-
A JSON string for selecting additional features.
Required: No
Type: Json
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*
Minimum:
0
Maximum:
10280
Update requires: Replacement
Applications
-
The applications to install on this cluster, for example, Spark, Flink, Oozie, Zeppelin, and so on.
Required: No
Type: Array of Application
Update requires: Replacement
AutoScalingRole
-
An IAM role for automatic scaling policies. The default role is
EMR_AutoScaling_DefaultRole
. The IAM role provides permissions that the automatic scaling feature requires to launch and terminate Amazon EC2 instances in an instance group.Required: No
Type: String
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*
Minimum:
0
Maximum:
10280
Update requires: Replacement
AutoTerminationPolicy
-
An auto-termination policy for an Amazon EMR cluster. An auto-termination policy defines the amount of idle time in seconds after which a cluster automatically terminates. For alternative cluster termination options, see Control cluster termination.
Required: No
Type: AutoTerminationPolicy
Update requires: No interruption
BootstrapActions
-
A list of bootstrap actions to run before Hadoop starts on the cluster nodes.
Required: No
Type: Array of BootstrapActionConfig
Update requires: Replacement
Configurations
-
Applies only to Amazon EMR releases 4.x and later. The list of configurations that are supplied to the Amazon EMR cluster.
Required: No
Type: Array of Configuration
Update requires: Replacement
CustomAmiId
-
Available only in Amazon EMR releases 5.7.0 and later. The ID of a custom Amazon EBS-backed Linux AMI if the cluster uses a custom AMI.
Required: No
Type: String
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*
Minimum:
0
Maximum:
256
Update requires: Replacement
EbsRootVolumeIops
-
The IOPS, of the Amazon EBS root device volume of the Linux AMI that is used for each Amazon EC2 instance. Available in Amazon EMR releases 6.15.0 and later.
Required: No
Type: Integer
Update requires: Replacement
EbsRootVolumeSize
-
The size, in GiB, of the Amazon EBS root device volume of the Linux AMI that is used for each Amazon EC2 instance. Available in Amazon EMR releases 4.x and later.
Required: No
Type: Integer
Update requires: Replacement
EbsRootVolumeThroughput
-
The throughput, in MiB/s, of the Amazon EBS root device volume of the Linux AMI that is used for each Amazon EC2 instance. Available in Amazon EMR releases 6.15.0 and later.
Required: No
Type: Integer
Update requires: Replacement
Instances
-
A specification of the number and type of Amazon EC2 instances.
Required: Yes
Type: JobFlowInstancesConfig
Update requires: No interruption
JobFlowRole
-
Also called instance profile and Amazon EC2 role. An IAM role for an Amazon EMR cluster. The Amazon EC2 instances of the cluster assume this role. The default role is
EMR_EC2_DefaultRole
. In order to use the default role, you must have already created it using the AWS CLI or console.Required: Yes
Type: String
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*
Minimum:
0
Maximum:
10280
Update requires: Replacement
KerberosAttributes
-
Attributes for Kerberos configuration when Kerberos authentication is enabled using a security configuration. For more information see Use Kerberos Authentication in the Amazon EMR Management Guide.
Required: No
Type: KerberosAttributes
Update requires: Replacement
LogEncryptionKmsKeyId
-
The AWS KMS key used for encrypting log files. This attribute is only available with Amazon EMR 5.30.0 and later, excluding Amazon EMR 6.0.0.
Required: No
Type: String
Update requires: Replacement
LogUri
-
The path to the Amazon S3 location where logs for this cluster are stored.
Required: No
Type: String
Update requires: Replacement
ManagedScalingPolicy
-
Creates or updates a managed scaling policy for an Amazon EMR cluster. The managed scaling policy defines the limits for resources, such as Amazon EC2 instances that can be added or terminated from a cluster. The policy only applies to the core and task nodes. The master node cannot be scaled after initial configuration.
Required: No
Type: ManagedScalingPolicy
Update requires: No interruption
Name
-
The name of the cluster. This parameter can't contain the characters <, >, $, |, or ` (backtick).
Required: Yes
Type: String
Update requires: Replacement
OSReleaseLabel
-
The Amazon Linux release specified in a cluster launch RunJobFlow request. If no Amazon Linux release was specified, the default Amazon Linux release is shown in the response.
Required: No
Type: String
Update requires: Replacement
PlacementGroupConfigs
Property description not available.
Required: No
Type: Array of PlacementGroupConfig
Update requires: Replacement
ReleaseLabel
-
The Amazon EMR release label, which determines the version of open-source application packages installed on the cluster. Release labels are in the form
emr-x.x.x
, where x.x.x is an Amazon EMR release version such asemr-5.14.0
. For more information about Amazon EMR release versions and included application versions and features, see https://docs.aws.amazon.com/emr/latest/ReleaseGuide/. The release label applies only to Amazon EMR releases version 4.0 and later. Earlier versions useAmiVersion
.Required: No
Type: String
Update requires: Replacement
ScaleDownBehavior
-
The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized.
TERMINATE_AT_INSTANCE_HOUR
indicates that Amazon EMR terminates nodes at the instance-hour boundary, regardless of when the request to terminate the instance was submitted. This option is only available with Amazon EMR 5.1.0 and later and is the default for clusters created using that version.TERMINATE_AT_TASK_COMPLETION
indicates that Amazon EMR adds nodes to a deny list and drains tasks from nodes before terminating the Amazon EC2 instances, regardless of the instance-hour boundary. With either behavior, Amazon EMR removes the least active nodes first and blocks instance termination if it could lead to HDFS corruption.TERMINATE_AT_TASK_COMPLETION
is available only in Amazon EMR releases 4.1.0 and later, and is the default for versions of Amazon EMR earlier than 5.1.0.Required: No
Type: String
Allowed values:
TERMINATE_AT_INSTANCE_HOUR | TERMINATE_AT_TASK_COMPLETION
Update requires: Replacement
SecurityConfiguration
-
The name of the security configuration applied to the cluster.
Required: No
Type: String
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*
Minimum:
0
Maximum:
10280
Update requires: Replacement
ServiceRole
-
The IAM role that Amazon EMR assumes in order to access AWS resources on your behalf.
Required: Yes
Type: String
Update requires: Replacement
StepConcurrencyLevel
-
Specifies the number of steps that can be executed concurrently. The default value is
1
. The maximum value is256
.Required: No
Type: Integer
Update requires: No interruption
Steps
-
A list of steps to run.
Required: No
Type: Array of StepConfig
Update requires: Replacement
-
A list of tags associated with a cluster.
Required: No
Type: Array of Tag
Update requires: No interruption
VisibleToAllUsers
-
Indicates whether the cluster is visible to all IAM users of the AWS account associated with the cluster. If this value is set to
true
, all IAM users of that AWS account can view and manage the cluster if they have the proper policy permissions set. If this value isfalse
, only the IAM user that created the cluster can view and manage it. This value can be changed using the SetVisibleToAllUsers action.Note
When you create clusters directly through the EMR console or API, this value is set to
true
by default. However, forAWS::EMR::Cluster
resources in CloudFormation, the default isfalse
.Required: No
Type: Boolean
Update requires: No interruption
Return values
Ref
When you pass the logical ID of this resource to the intrinsic Ref
function, Ref
returns returns the cluster ID, such as j-1ABCD123AB1A.
For more information about using the Ref
function, see Ref
.
Fn::GetAtt
The Fn::GetAtt
intrinsic function returns a value for a specified attribute of this type. The following are the available attributes and sample return values.
For more information about using the Fn::GetAtt
intrinsic function, see Fn::GetAtt
.
Id
-
The unique identifier for the cluster.
MasterPublicDNS
-
The public DNS name of the master node (instance), such as
ec2-12-123-123-123.us-west-2.compute.amazonaws.com
.
Examples
Create cluster examples.
Create a cluster using a custom AMI for EC2 instances
The following example template specifies a cluster using a custom Amazon Linux AMI for the EC2 instances in the cluster.
JSON
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters" : {
"CustomAmiId" : {
"Type" : "String"
},
"InstanceType" : {
"Type" : "String"
},
"ReleaseLabel" : {
"Type" : "String"
},
"SubnetId" : {
"Type" : "String"
},
"TerminationProtected" : {
"Type" : "String",
"Default" : "false"
},
"ElasticMapReducePrincipal" : {
"Type" : "String"
},
"Ec2Principal" : {
"Type" : "String"
}
},
"Resources": {
"cluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"CustomAmiId" : {"Ref" : "CustomAmiId"},
"Instances": {
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnMaster"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnCore"
},
"TaskInstanceGroups": [
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-1"
},
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-2"
}
],
"TerminationProtected" : {"Ref" : "TerminationProtected"},
"Ec2SubnetId" : {"Ref" : "SubnetId"}
},
"Name": "CFNtest",
"JobFlowRole" : {"Ref": "emrEc2InstanceProfile"},
"ServiceRole" : {"Ref": "emrRole"},
"ReleaseLabel" : {"Ref" : "ReleaseLabel"},
"VisibleToAllUsers" : true,
"Tags": [
{
"Key": "key1",
"Value": "value1"
}
]
}
},
"emrRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": {"Ref" : "ElasticMapReducePrincipal"}
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"]
}
},
"emrEc2Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": {"Ref" : "Ec2Principal"}
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"]
}
},
"emrEc2InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ {
"Ref": "emrEc2Role"
} ]
}
}
}
}
YAML
AWSTemplateFormatVersion: 2010-09-09
Parameters:
CustomAmiId:
Type: String
InstanceType:
Type: String
ReleaseLabel:
Type: String
SubnetId:
Type: String
TerminationProtected:
Type: String
Default: 'false'
ElasticMapReducePrincipal:
Type: String
Ec2Principal:
Type: String
Resources:
cluster:
Type: AWS::EMR::Cluster
Properties:
CustomAmiId: !Ref CustomAmiId
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnMaster
CoreInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnCore
TaskInstanceGroups:
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-1
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-2
TerminationProtected: !Ref TerminationProtected
Ec2SubnetId: !Ref SubnetId
Name: CFNtest
JobFlowRole: !Ref emrEc2InstanceProfile
ServiceRole: !Ref emrRole
ReleaseLabel: !Ref ReleaseLabel
VisibleToAllUsers: true
Tags:
- Key: key1
Value: value1
emrRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: !Ref ElasticMapReducePrincipal
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole'
emrEc2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: !Ref Ec2Principal
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role'
emrEc2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref emrEc2Role
Create a cluster and specify the root volume size of EC2 instances
The following example template enables you to specify the size of the EBS root volume for cluster instances.
JSON
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters" : {
"InstanceType" : {
"Type" : "String"
},
"ReleaseLabel" : {
"Type" : "String"
},
"SubnetId" : {
"Type" : "String"
},
"TerminationProtected" : {
"Type" : "String",
"Default" : "false"
},
"EbsRootVolumeSize" : {
"Type" : "String"
}
},
"Resources": {
"cluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"EbsRootVolumeSize" : {"Ref" : "EbsRootVolumeSize"},
"Instances": {
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnMaster"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnCore"
},
"TaskInstanceGroups": [
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-1"
},
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-2"
}
],
"TerminationProtected" : {"Ref" : "TerminationProtected"},
"Ec2SubnetId" : {"Ref" : "SubnetId"}
},
"Name": "CFNtest",
"JobFlowRole" : {"Ref": "emrEc2InstanceProfile"},
"ServiceRole" : {"Ref": "emrRole"},
"ReleaseLabel" : {"Ref" : "ReleaseLabel"},
"VisibleToAllUsers" : true,
"Tags": [
{
"Key": "key1",
"Value": "value1"
}
]
}
},
"emrRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "elasticmapreduce.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"]
}
},
"emrEc2Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"]
}
},
"emrEc2InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ {
"Ref": "emrEc2Role"
} ]
}
}
}
}
YAML
AWSTemplateFormatVersion: 2010-09-09
Parameters:
InstanceType:
Type: String
ReleaseLabel:
Type: String
SubnetId:
Type: String
TerminationProtected:
Type: String
Default: 'false'
EbsRootVolumeSize:
Type: String
Resources:
cluster:
Type: AWS::EMR::Cluster
Properties:
EbsRootVolumeSize: !Ref EbsRootVolumeSize
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnMaster
CoreInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnCore
TaskInstanceGroups:
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-1
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-2
TerminationProtected: !Ref TerminationProtected
Ec2SubnetId: !Ref SubnetId
Name: CFNtest
JobFlowRole: !Ref emrEc2InstanceProfile
ServiceRole: !Ref emrRole
ReleaseLabel: !Ref ReleaseLabel
VisibleToAllUsers: true
Tags:
- Key: key1
Value: value1
emrRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: elasticmapreduce.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole'
emrEc2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role'
emrEc2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref emrEc2Role
Create a cluster with Kerberos authentication
The following example template enables you to specify the Kerberos authentication configuration for an EMR cluster.
JSON
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters" : {
"CrossRealmTrustPrincipalPassword" : {
"Type" : "String"
},
"KdcAdminPassword" : {
"Type" : "String"
},
"Realm" : {
"Type" : "String"
},
"InstanceType" : {
"Type" : "String"
},
"ReleaseLabel" : {
"Type" : "String"
},
"SubnetId" : {
"Type" : "String"
}
},
"Resources": {
"cluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Instances": {
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnMaster"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnCore"
},
"TaskInstanceGroups": [
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-1"
},
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-2"
}
],
"Ec2SubnetId" : {"Ref" : "SubnetId"}
},
"Name": "CFNtest2",
"JobFlowRole" : {"Ref": "emrEc2InstanceProfile"},
"KerberosAttributes" : {
"CrossRealmTrustPrincipalPassword" : "CfnIntegrationTest-1",
"KdcAdminPassword" : "CfnIntegrationTest-1",
"Realm": "EC2.INTERNAL"
},
"ServiceRole" : {"Ref": "emrRole"},
"ReleaseLabel" : {"Ref" : "ReleaseLabel"},
"SecurityConfiguration" : {"Ref" : "securityConfiguration"},
"VisibleToAllUsers" : true,
"Tags": [
{
"Key": "key1",
"Value": "value1"
}
]
}
},
"key" : {
"Type" : "AWS::KMS::Key",
"Properties" : {
"KeyPolicy" : {
"Version": "2012-10-17",
"Id": "key-default-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": { "Fn::GetAtt" : ["emrEc2Role", "Arn"]}
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": { "Fn::Join" : ["" , ["arn:aws:iam::", {"Ref" : "AWS::AccountId"} ,":root" ]] }
},
"Action": "kms:*",
"Resource": "*"
}
]
}
}
},
"securityConfiguration": {
"Type" : "AWS::EMR::SecurityConfiguration",
"Properties" : {
"SecurityConfiguration" : {
"AuthenticationConfiguration": {
"KerberosConfiguration": {
"Provider": "ClusterDedicatedKdc",
"ClusterDedicatedKdcConfiguration": {
"TicketLifetimeInHours": 24,
"CrossRealmTrustConfiguration": {
"Realm": "AD.DOMAIN.COM",
"Domain": "ad.domain.com",
"AdminServer": "ad.domain.com",
"KdcServer": "ad.domain.com"
}
}
}
}
}
}
},
"emrRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "elasticmapreduce.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"]
}
},
"emrEc2Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"]
}
},
"emrEc2InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ {
"Ref": "emrEc2Role"
} ]
}
}
},
"Outputs" : {
"keyArn" : {
"Value" : {"Fn::GetAtt" : ["key", "Arn"]}
}
}
}
YAML
AWSTemplateFormatVersion: 2010-09-09
Parameters:
CrossRealmTrustPrincipalPassword:
Type: String
KdcAdminPassword:
Type: String
Realm:
Type: String
InstanceType:
Type: String
ReleaseLabel:
Type: String
SubnetId:
Type: String
Resources:
cluster:
Type: 'AWS::EMR::Cluster'
Properties:
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnMaster
CoreInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnCore
TaskInstanceGroups:
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-1
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-2
Ec2SubnetId: !Ref SubnetId
Name: CFNtest2
JobFlowRole: !Ref emrEc2InstanceProfile
KerberosAttributes:
CrossRealmTrustPrincipalPassword: CfnIntegrationTest-1
KdcAdminPassword: CfnIntegrationTest-1
Realm: EC2.INTERNAL
ServiceRole: !Ref emrRole
ReleaseLabel: !Ref ReleaseLabel
SecurityConfiguration: !Ref securityConfiguration
VisibleToAllUsers: true
Tags:
- Key: key1
Value: value1
key:
Type: 'AWS::KMS::Key'
Properties:
KeyPolicy:
Version: 2012-10-17
Id: key-default-1
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !GetAtt
- emrEc2Role
- Arn
Action: 'kms:*'
Resource: '*'
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Join
- ''
- - 'arn:aws:iam::'
- !Ref 'AWS::AccountId'
- ':root'
Action: 'kms:*'
Resource: '*'
securityConfiguration:
Type: 'AWS::EMR::SecurityConfiguration'
Properties:
SecurityConfiguration:
AuthenticationConfiguration:
KerberosConfiguration:
Provider: ClusterDedicatedKdc
ClusterDedicatedKdcConfiguration:
TicketLifetimeInHours: 24
CrossRealmTrustConfiguration:
Realm: AD.DOMAIN.COM
Domain: ad.domain.com
AdminServer: ad.domain.com
KdcServer: ad.domain.com
emrRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: elasticmapreduce.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole'
emrEc2Role:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role'
emrEc2InstanceProfile:
Type: 'AWS::IAM::InstanceProfile'
Properties:
Path: /
Roles:
- !Ref emrEc2Role
Outputs:
keyArn:
Value: !GetAtt
- key
- Arn
Create a cluster with a managed scaling policy
The following example template enables you to specify the managed scaling policy for an EMR cluster.
JSON
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters" : {
"InstanceType" : {
"Type" : "String"
},
"ReleaseLabel" : {
"Type" : "String"
},
"SubnetId" : {
"Type" : "String"
},
"MinimumCapacityUnits" : {
"Type" : "String"
},
"MaximumCapacityUnits" : {
"Type" : "String"
},
"MaximumCoreCapacityUnits" : {
"Type" : "String"
},
"MaximumOnDemandCapacityUnits" : {
"Type" : "String"
},
"UnitType" : {
"Type" : "String"
}
},
"Resources": {
"cluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Instances": {
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "primary"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "core"
},
"TaskInstanceGroups": [
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-1"
},
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-2"
}
],
"Ec2SubnetId" : {"Ref" : "SubnetId"}
},
"Name": "ManagedScalingExample",
"JobFlowRole" : "EMR_EC2_DefaultRole",
"ServiceRole" : "EMR_DefaultRole",
"ReleaseLabel" : {"Ref" : "ReleaseLabel"},
"ManagedScalingPolicy" : {
"ComputeLimits" : {
"MinimumCapacityUnits" : {"Ref": "MinimumCapacityUnits"},
"MaximumCapacityUnits" : {"Ref": "MaximumCapacityUnits"},
"MaximumCoreCapacityUnits" : {"Ref": "MaximumCoreCapacityUnits"},
"MaximumOnDemandCapacityUnits" : {"Ref": "MaximumOnDemandCapacityUnits"},
"UnitType" : {"Ref": "UnitType"}
}
}
}
}
}
}
YAML
AWSTemplateFormatVersion: 2010-09-09
Parameters:
InstanceType:
Type: String
ReleaseLabel:
Type: String
SubnetId:
Type: String
MinimumCapacityUnits:
Type: String
MaximumCapacityUnits:
Type: String
MaximumCoreCapacityUnits:
Type: String
MaximumOnDemandCapacityUnits:
Type: String
UnitType:
Type: String
Resources:
cluster:
Type: 'AWS::EMR::Cluster'
Properties:
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: primary
CoreInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: core
TaskInstanceGroups:
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-1
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-2
Ec2SubnetId: !Ref SubnetId
Name: ManagedScalingExample
JobFlowRole: EMR_EC2_DefaultRole
ServiceRole: EMR_DefaultRole
ReleaseLabel: !Ref ReleaseLabel
ManagedScalingPolicy:
ComputeLimits:
MinimumCapacityUnits: !Ref MinimumCapacityUnits
MaximumCapacityUnits: !Ref MaximumCapacityUnits
MaximumCoreCapacityUnits: !Ref MaximumCoreCapacityUnits
MaximumOnDemandCapacityUnits: !Ref MaximumOnDemandCapacityUnits
UnitType: !Ref UnitType
Create a cluster with task instance groups
The following example template enables you to create task instance groups for an EMR cluster.
JSON
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters" : {
"InstanceType" : {
"Type" : "String"
},
"ReleaseLabel" : {
"Type" : "String"
},
"SubnetId" : {
"Type" : "String"
},
"TerminationProtected" : {
"Type" : "String",
"Default" : "false"
},
"ElasticMapReducePrincipal" : {
"Type" : "String"
},
"Ec2Principal" : {
"Type" : "String"
}
},
"Resources": {
"cluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Instances": {
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnMaster"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnCore"
},
"TaskInstanceGroups": [
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-1"
},
{
"InstanceCount": 1,
"InstanceType": {"Ref" : "InstanceType"},
"Market": "ON_DEMAND",
"Name": "cfnTask-2"
}
],
"TerminationProtected" : {"Ref" : "TerminationProtected"},
"Ec2SubnetId" : {"Ref" : "SubnetId"}
},
"Name": "CFNtest",
"JobFlowRole" : {"Ref": "emrEc2InstanceProfile"},
"ServiceRole" : {"Ref": "emrRole"},
"ReleaseLabel" : {"Ref" : "ReleaseLabel"},
"VisibleToAllUsers" : true,
"Tags": [
{
"Key": "key1",
"Value": "value1"
}
]
}
},
"emrRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": {"Ref" : "ElasticMapReducePrincipal"}
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"]
}
},
"emrEc2Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": {"Ref" : "Ec2Principal"}
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"]
}
},
"emrEc2InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ {
"Ref": "emrEc2Role"
} ]
}
}
}
}
YAML
AWSTemplateFormatVersion: 2010-09-09
Parameters:
InstanceType:
Type: String
ReleaseLabel:
Type: String
SubnetId:
Type: String
TerminationProtected:
Type: String
Default: 'false'
ElasticMapReducePrincipal:
Type: String
Ec2Principal:
Type: String
Resources:
cluster:
Type: AWS::EMR::Cluster
Properties:
Instances:
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnMaster
CoreInstanceGroup:
InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnCore
TaskInstanceGroups:
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-1
- InstanceCount: 1
InstanceType: !Ref InstanceType
Market: ON_DEMAND
Name: cfnTask-2
TerminationProtected: !Ref TerminationProtected
Ec2SubnetId: !Ref SubnetId
Name: CFNtest
JobFlowRole: !Ref emrEc2InstanceProfile
ServiceRole: !Ref emrRole
ReleaseLabel: !Ref ReleaseLabel
VisibleToAllUsers: true
Tags:
- Key: key1
Value: value1
emrRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: !Ref ElasticMapReducePrincipal
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole'
emrEc2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: !Ref Ec2Principal
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role'
emrEc2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref emrEc2Role
Create a cluster with a task instance fleet
The following example template enables you to create a task instance fleet for an EMR cluster.
JSON
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters" : {
"InstanceType" : {
"Type" : "String"
},
"ReleaseLabel" : {
"Type" : "String"
},
"SubnetId" : {
"Type" : "String"
},
"TerminationProtected" : {
"Type" : "String",
"Default" : "false"
},
"ElasticMapReducePrincipal" : {
"Type" : "String"
},
"Ec2Principal" : {
"Type" : "String"
}
},
"Resources": {
"cluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Instances": {
"MasterInstanceFleet": {
"Name": "cfnMaster",
"TargetOnDemandCapacity": 1,
"TargetSpotCapacity": 0,
"InstanceTypeConfigs": [
{
"InstanceType": {"Ref" : "InstanceType"},
"WeightedCapacity": 1
}
]
},
"CoreInstanceFleet": {
"Name": "cfnCore",
"TargetOnDemandCapacity": 1,
"TargetSpotCapacity": 0,
"InstanceTypeConfigs": [
{
"InstanceType": {"Ref" : "InstanceType"},
"WeightedCapacity": 1
}
]
},
"TaskInstanceFleets": [
{
"Name": "cfnTask",
"TargetOnDemandCapacity": 1,
"TargetSpotCapacity": 0,
"InstanceTypeConfigs": [
{
"InstanceType": {"Ref" : "InstanceType"},
"WeightedCapacity": 1
}
]
}
],
"TerminationProtected" : {"Ref" : "TerminationProtected"},
"Ec2SubnetIds" : [
{"Ref" : "SubnetId"}
]
},
"Name": "CFNtest",
"JobFlowRole" : {"Ref": "emrEc2InstanceProfile"},
"ServiceRole" : {"Ref": "emrRole"},
"ReleaseLabel" : {"Ref" : "ReleaseLabel"},
"VisibleToAllUsers" : true,
"Tags": [
{
"Key": "key1",
"Value": "value1"
}
]
}
},
"emrRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": {"Ref" : "ElasticMapReducePrincipal"}
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"]
}
},
"emrEc2Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": {"Ref" : "Ec2Principal"}
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"]
}
},
"emrEc2InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ {
"Ref": "emrEc2Role"
} ]
}
}
}
}
YAML
AWSTemplateFormatVersion: 2010-09-09
Parameters:
InstanceType:
Type: String
ReleaseLabel:
Type: String
SubnetId:
Type: String
TerminationProtected:
Type: String
Default: 'false'
ElasticMapReducePrincipal:
Type: String
Ec2Principal:
Type: String
Resources:
cluster:
Type: AWS::EMR::Cluster
Properties:
Instances:
MasterInstanceFleet:
Name: cfnMaster
TargetOnDemandCapacity: 1
TargetSpotCapacity: 0
InstanceTypeConfigs:
- InstanceType: !Ref InstanceType
WeightedCapacity: 1
CoreInstanceFleet:
Name: cfnCore
TargetOnDemandCapacity: 1
TargetSpotCapacity: 0
InstanceTypeConfigs:
- InstanceType: !Ref InstanceType
WeightedCapacity: 1
TaskInstanceFleets:
- Name: cfnTask
TargetOnDemandCapacity: 1
TargetSpotCapacity: 0
InstanceTypeConfigs:
- InstanceType: !Ref InstanceType
WeightedCapacity: 1
TerminationProtected: !Ref TerminationProtected
Ec2SubnetIds:
- !Ref SubnetId
Name: CFNtest
JobFlowRole: !Ref emrEc2InstanceProfile
ServiceRole: !Ref emrRole
ReleaseLabel: !Ref ReleaseLabel
VisibleToAllUsers: true
Tags:
- Key: key1
Value: value1
emrRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: !Ref ElasticMapReducePrincipal
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole'
emrEc2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2008-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: !Ref Ec2Principal
Action: 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role'
emrEc2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref emrEc2Role