AWS::EMR::InstanceGroupConfig - AWS CloudFormation
SyntaxPropertiesReturn valuesExamples

This is the new AWS CloudFormation Template Reference Guide. Please update your bookmarks and links. For help getting started with CloudFormation, see the AWS CloudFormation User Guide.

AWS::EMR::InstanceGroupConfig

Use InstanceGroupConfig to define instance groups for an EMR cluster. A cluster can not use both instance groups and instance fleets. For more information, see Create a Cluster with Instance Fleets or Uniform Instance Groups in the Amazon EMR Management Guide.

Important

You can currently only add task instance groups to a cluster with this resource. If you use this resource, CloudFormation waits for the cluster launch to complete before adding the task instance group to the cluster. In order to add task instance groups to the cluster as part of the cluster launch and minimize delays in provisioning task nodes, use the TaskInstanceGroups subproperty for the AWS::EMR::Cluster JobFlowInstancesConfig property instead. To use this subproperty, see AWS::EMR::Cluster for examples.

Syntax

To declare this entity in your AWS CloudFormation template, use the following syntax:

JSON

{
  "Type" : "AWS::EMR::InstanceGroupConfig",
  "Properties" : {
      "AutoScalingPolicy" : AutoScalingPolicy,
      "BidPrice" : String,
      "Configurations" : [ Configuration, ... ],
      "CustomAmiId" : String,
      "EbsConfiguration" : EbsConfiguration,
      "InstanceCount" : Integer,
      "InstanceRole" : String,
      "InstanceType" : String,
      "JobFlowId" : String,
      "Market" : String,
      "Name" : String
    }
}

YAML

Type: AWS::EMR::InstanceGroupConfig
Properties:
  AutoScalingPolicy: 
    AutoScalingPolicy
  BidPrice: String
  Configurations: 
    - Configuration
  CustomAmiId: String
  EbsConfiguration: 
    EbsConfiguration
  InstanceCount: Integer
  InstanceRole: String
  InstanceType: String
  JobFlowId: String
  Market: String
  Name: String

Properties

AutoScalingPolicy

AutoScalingPolicy is a subproperty of InstanceGroupConfig. AutoScalingPolicy defines how an instance group dynamically adds and terminates EC2 instances in response to the value of a CloudWatch metric. For more information, see Using Automatic Scaling in Amazon EMR in the Amazon EMR Management Guide.

Required: No

Type: AutoScalingPolicy

Update requires: No interruption

BidPrice

If specified, indicates that the instance group uses Spot Instances. This is the maximum price you are willing to pay for Spot Instances. Specify OnDemandPrice to set the amount equal to the On-Demand price, or specify an amount in USD.

Required: No

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 0

Maximum: 256

Update requires: Replacement

Configurations
Note

Amazon EMR releases 4.x or later.

The list of configurations supplied for an Amazon EMR cluster instance group. You can specify a separate configuration for each instance group (master, core, and task).

Required: No

Type: Array of Configuration

Update requires: Replacement

CustomAmiId

The custom AMI ID to use for the provisioned instance group.

Required: No

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 0

Maximum: 256

Update requires: Replacement

EbsConfiguration

EbsConfiguration determines the EBS volumes to attach to EMR cluster instances.

Required: No

Type: EbsConfiguration

Update requires: Replacement

InstanceCount

Target number of instances for the instance group.

Required: Yes

Type: Integer

Update requires: No interruption

InstanceRole

The role of the instance group in the cluster.

Allowed Values: TASK

Required: Yes

Type: String

Update requires: Replacement

InstanceType

The Amazon EC2 instance type for all instances in the instance group.

Required: Yes

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 1

Maximum: 256

Update requires: Replacement

JobFlowId

The ID of an Amazon EMR cluster that you want to associate this instance group with.

Required: Yes

Type: String

Update requires: Replacement

Market

Market type of the Amazon EC2 instances used to create a cluster node.

Required: No

Type: String

Allowed values: ON_DEMAND | SPOT

Update requires: Replacement

Name

Friendly name given to the instance group.

Required: No

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Minimum: 0

Maximum: 256

Update requires: Replacement

Return values

Ref

When you pass the logical ID of this resource to the intrinsic Ref function, Ref returns returns the ID of the instance group.

For more information about using the Ref function, see Ref.

Fn::GetAtt

Examples

Example instance group configuration specifications.

Add a task instance group

The following example adds a task instance group to a cluster named TestCluster.

JSON

"TestInstanceGroupConfig": {
    "Type": "AWS: : EMR: : InstanceGroupConfig",
    "Properties": {
        "InstanceCount": 2,
        "InstanceType": "m3.xlarge",
        "InstanceRole": "TASK",
        "Market": "ON_DEMAND",
        "Name": "cfnTask2",
        "JobFlowId": {
            "Ref": "cluster"
        }
    }
}

YAML

TestInstanceGroupConfig: 
  Properties: 
    InstanceCount: 2
    InstanceRole: TASK
    InstanceType: m3.xlarge
    JobFlowId: 
      Ref: cluster
    Market: ON_DEMAND
    Name: cfnTask2
  Type: "AWS::EMR::InstanceGroupConfig"

Specify an Automatic Scaling Policy

The following example defines an AutoScalingPolicy for an InstanceGroupConfig resource.

JSON

"MyInstanceGroupConfig": {
  "Type": "AWS::EMR::InstanceGroupConfig",
  "Properties": {
    "InstanceCount": 1,
    "InstanceType": {
      "Ref": "InstanceType"
    },
    "InstanceRole": "TASK",
    "Market": "ON_DEMAND",
    "Name": "cfnTask",
    "JobFlowId": {
      "Ref": "MyCluster"
    },
    "AutoScalingPolicy": {
      "Constraints": {
        "MinCapacity": {
          "Ref": "MinCapacity"
        },
        "MaxCapacity": {
          "Ref": "MaxCapacity"
        }
      },
      "Rules": [
        {
          "Name": "Scale-out",
          "Description": "Scale-out policy",
          "Action": {
            "SimpleScalingPolicyConfiguration": {
              "AdjustmentType": "CHANGE_IN_CAPACITY",
              "ScalingAdjustment": 1,
              "CoolDown": 300
            }
          },
          "Trigger": {
            "CloudWatchAlarmDefinition": {
              "Dimensions": [
                {
                  "Key": "JobFlowId",
                  "Value": "${emr.clusterId}"
                }
              ],
              "EvaluationPeriods": 1,
              "Namespace": "AWS/ElasticMapReduce",
              "Period": 300,
              "ComparisonOperator": "LESS_THAN",
              "Statistic": "AVERAGE",
              "Threshold": 15,
              "Unit": "PERCENT",
              "MetricName": "YARNMemoryAvailablePercentage"
            }
          }
        },
        {
          "Name": "Scale-in",
          "Description": "Scale-in policy",
          "Action": {
            "SimpleScalingPolicyConfiguration": {
              "AdjustmentType": "CHANGE_IN_CAPACITY",
              "ScalingAdjustment": -1,
              "CoolDown": 300
            }
          },
          "Trigger": {
            "CloudWatchAlarmDefinition": {
              "Dimensions": [
                {
                  "Key": "JobFlowId",
                  "Value": "${emr.clusterId}"
                }
              ],
              "EvaluationPeriods": 1,
              "Namespace": "AWS/ElasticMapReduce",
              "Period": 300,
              "ComparisonOperator": "GREATER_THAN",
              "Statistic": "AVERAGE",
              "Threshold": 75,
              "Unit": "PERCENT",
              "MetricName": "YARNMemoryAvailablePercentage"
            }
          }
        }
      ]
    }
  }
}

YAML

MyInstanceGroupConfig:
  Type: 'AWS::EMR::InstanceGroupConfig'
  Properties:
    InstanceCount: 1
    InstanceType: !Ref InstanceType
    InstanceRole: TASK
    Market: ON_DEMAND
    Name: cfnTask
    JobFlowId: !Ref MyCluster
    AutoScalingPolicy:
      Constraints:
        MinCapacity: !Ref MinCapacity
        MaxCapacity: !Ref MaxCapacity
      Rules:
        - Name: Scale-out
          Description: Scale-out policy
          Action:
            SimpleScalingPolicyConfiguration:
              AdjustmentType: CHANGE_IN_CAPACITY
              ScalingAdjustment: 1
              CoolDown: 300
          Trigger:
            CloudWatchAlarmDefinition:
              Dimensions:
                - Key: JobFlowId
                  Value: '${emr.clusterId}'
              EvaluationPeriods: 1
              Namespace: AWS/ElasticMapReduce
              Period: 300
              ComparisonOperator: LESS_THAN
              Statistic: AVERAGE
              Threshold: 15
              Unit: PERCENT
              MetricName: YARNMemoryAvailablePercentage
        - Name: Scale-in
          Description: Scale-in policy
          Action:
            SimpleScalingPolicyConfiguration:
              AdjustmentType: CHANGE_IN_CAPACITY
              ScalingAdjustment: -1
              CoolDown: 300
          Trigger:
            CloudWatchAlarmDefinition:
              Dimensions:
                - Key: JobFlowId
                  Value: '${emr.clusterId}'
              EvaluationPeriods: 1
              Namespace: AWS/ElasticMapReduce
              Period: 300
              ComparisonOperator: GREATER_THAN
              Statistic: AVERAGE
              Threshold: 75
              Unit: PERCENT
              MetricName: YARNMemoryAvailablePercentage