# Control network traffic with security groups for your Amazon EMR cluster Security groups act as virtual firewalls for EC2 instances in your cluster to control inbound and outbound traffic. Each security group has a set of rules that control inbound traffic, and a separate set of rules to control outbound traffic. For more information, see [Amazon EC2 security groups for Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html) in the *Amazon EC2 User Guide*. You use two classes of security groups with Amazon EMR: *Amazon EMR-managed security groups* and *additional security groups*. Every cluster has managed security groups associated with it. You can use the default managed security groups that Amazon EMR creates, or specify custom managed security groups. Either way, Amazon EMR automatically adds rules to managed security groups that a cluster needs to communicate between cluster instances and AWS services. Additional security groups are optional. You can specify them in addition to managed security groups to tailor access to cluster instances. Additional security groups contain only rules that you define. Amazon EMR does not modify them. The rules that Amazon EMR creates in managed security groups allow the cluster to communicate among internal components. To allow users and applications to access a cluster from outside the cluster, you can edit rules in managed security groups, you can create additional security groups with additional rules, or do both. **Important** Editing rules in managed security groups may have unintended consequences. You may inadvertently block the traffic required for clusters to function properly and cause errors because nodes are unreachable. Carefully plan and test security group configurations before implementation. You can specify security groups only when you create a cluster. They can't be added to a cluster or cluster instances while a cluster is running, but you can edit, add, and remove rules from existing security groups. The rules take effect as soon as you save them. Security groups are restrictive by default. Unless a rule is added that allows traffic, the traffic is rejected. If there is more than one rule that applies to the same traffic and the same source, the most permissive rule applies. For example, if you have a rule that allows SSH from IP address 192.0.2.12/32, and another rule that allows access to all TCP traffic from the range 192.0.2.0/24, the rule that allows all TCP traffic from the range that includes 192.0.2.12 takes precedence. In this case, the client at 192.0.2.12 might have more access than you intended. **Important** Use caution when you edit security group rules to open ports. Be sure to add rules that only allow traffic from trusted and authenticated clients for the protocols and ports that are required to run your workloads. You can configure Amazon EMR *block public access* in each Region that you use to prevent cluster creation if a rule allows public access on any port that you don't add to a list of exceptions. For AWS accounts created after July 2019, Amazon EMR block public access is on by default. For AWS accounts that created a cluster before July 2019, Amazon EMR block public access is off by default. For more information, see [Using Amazon EMR block public access](emr-block-public-access.md). **Topics** + [Working with Amazon EMR-managed security groups](emr-man-sec-groups.md) + [Working with additional security groups for an Amazon EMR cluster](emr-additional-sec-groups.md) + [Specifying Amazon EMR-managed and additional security groups](emr-sg-specify.md) + [Specifying EC2 security groups for EMR Notebooks](emr-managed-notebooks-security-groups.md) + [Using Amazon EMR block public access](emr-block-public-access.md) **Note** Amazon EMR aims to use inclusive alternatives for potentially offensive or non-inclusive industry terms such as "master" and "slave". We've transitioned to new terminology to foster a more inclusive experience and to facilitate your understanding of the service components. We now describe "nodes" as **instances**, and we describe Amazon EMR instance types as **primary**, **core**, and **task** instances. During the transition, you might still find legacy references to the outdated terms, such as those that pertain to security groups for Amazon EMR. # Working with Amazon EMR-managed security groups **Note** Amazon EMR aims to use inclusive alternatives for potentially offensive or non-inclusive industry terms such as "master" and "slave". We've transitioned to new terminology to foster a more inclusive experience and to facilitate your understanding of the service components. We now describe "nodes" as **instances**, and we describe Amazon EMR instance types as **primary**, **core**, and **task** instances. During the transition, you might still find legacy references to the outdated terms, such as those that pertain to security groups for Amazon EMR. Different managed security groups are associated with the primary instance and with the core and task instances in a cluster. An additional managed security group for service access is required when you create a cluster in a private subnet. For more information about the role of managed security groups with respect to your network configuration, see [Amazon VPC options when you launch a cluster](emr-clusters-in-a-vpc.md). When you specify managed security groups for a cluster, you must use the same type of security group, default or custom, for all managed security groups. For example, you can't specify a custom security group for the primary instance, and then not specify a custom security group for core and task instances. If you use default managed security groups, you don't need to specify them when you create a cluster. Amazon EMR automatically uses the defaults. Moreover, if the defaults don't exist in the cluster's VPC yet, Amazon EMR creates them. Amazon EMR also creates them if you explicitly specify them and they don't exist yet. You can edit rules in managed security groups after clusters are created. When you create a new cluster, Amazon EMR checks the rules in the managed security groups that you specify, and then creates any missing *inbound* rules that the new cluster needs in addition to rules that may have been added earlier. Unless specifically stated otherwise, each rule for default Amazon EMR-managed security groups is also added to custom Amazon EMR-managed security groups that you specify. The default managed security groups are as follows: + **ElasticMapReduce-primary** For rules in this security group, see [Amazon EMR-managed security group for the primary instance (public subnets)](#emr-sg-elasticmapreduce-master). + **ElasticMapReduce-core** For rules in this security group, see [Amazon EMR-managed security group for core and task instances (public subnets)](#emr-sg-elasticmapreduce-slave). + **ElasticMapReduce-Primary-Private** For rules in this security group, see [Amazon EMR-managed security group for the primary instance (private subnets)](#emr-sg-elasticmapreduce-master-private). + **ElasticMapReduce-Core-Private** For rules in this security group, see [Amazon EMR-managed security group for core and task instances (private subnets)](#emr-sg-elasticmapreduce-slave-private). + **ElasticMapReduce-ServiceAccess** For rules in this security group, see [Amazon EMR-managed security group for service access (private subnets)](#emr-sg-elasticmapreduce-sa-private). ## Amazon EMR-managed security group for the primary instance (public subnets) The default managed security group for the primary instance in public subnets has the **Group Name** of **ElasticMapReduce-primary**. It has the following rules. If you specify a custom managed security group, Amazon EMR adds all the same rules to your custom security group. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-man-sec-groups.html) **To grant trusted sources SSH access to the primary security group with the console** To edit your security groups, you must have permission to manage security groups for the VPC that the cluster is in. For more information, see [Changing Permissions for a user](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_users_change-permissions.html) and the [Example Policy](https://docs.aws.amazon.com//IAM/latest/UserGuide/reference_policies_examples_ec2_securitygroups-vpc.html) that allows managing EC2 security groups in the *IAM User Guide*. 1. Sign in to the AWS Management Console, and open the Amazon EMR console at [https://console.aws.amazon.com/emr](https://console.aws.amazon.com/emr). 1. Choose **Clusters**. Choose the **ID** of the cluster you want to modify. 1. In the **Network and security** pane, expand the **EC2 security groups (firewall)** dropdown. 1. Under **Primary node**, choose your security group. 1. Choose **Edit inbound rules**. 1. Check for an inbound rule that allows public access with the following settings. If it exists, choose **Delete** to remove it. + **Type** SSH + **Port** 22 + **Source** Custom 0.0.0.0/0 **Warning** Before December 2020, there was a pre-configured rule to allow inbound traffic on Port 22 from all sources. This rule was created to simplify initial SSH connections to the primary node. We strongly recommend that you remove this inbound rule and restrict traffic to trusted sources. 1. Scroll to the bottom of the list of rules and choose **Add Rule**. 1. For **Type**, select **SSH**. Selecting SSH automatically enters **TCP** for **Protocol** and **22** for **Port Range**. 1. For source, select **My IP** to automatically add your IP address as the source address. You can also add a range of **Custom** trusted client IP addresses, or create additional rules for other clients. Many network environments dynamically allocate IP addresses, so you might need to update your IP addresses for trusted clients in the future. 1. Choose **Save**. 1. Optionally, choose the other security group under **Core and task nodes** in the **Network and security ** pane and repeat the steps above to allow SSH client access to core and task nodes. ## Amazon EMR-managed security group for core and task instances (public subnets) The default managed security group for core and task instances in public subnets has the **Group Name** of **ElasticMapReduce-core**. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-man-sec-groups.html) ## Amazon EMR-managed security group for the primary instance (private subnets) The default managed security group for the primary instance in private subnets has the **Group Name** of **ElasticMapReduce-Primary-Private**. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-man-sec-groups.html) ## Amazon EMR-managed security group for core and task instances (private subnets) The default managed security group for core and task instances in private subnets has the **Group Name** of **ElasticMapReduce-Core-Private**. The default managed security group has the following rules, and Amazon EMR adds the same rules if you specify a custom managed security group. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-man-sec-groups.html) ### Editing outbound rules By default, Amazon EMR creates this security group with outbound rules that allow all outbound traffic on all protocols and ports. Allowing all outbound traffic is selected because various Amazon EMR and customer applications that can run on Amazon EMR clusters may require different egress rules. Amazon EMR is not able to anticipate these specific settings when creating default security groups. You can scope down egress in your security groups to include only those rules that suit your use cases and security policies. At minimum, this security group requires the following outbound rules, but some applications might need additional egress. | Type | Protocol | Port range | Destination | Details | | --- | --- | --- | --- | --- | | All TCP | TCP | All | pl-xxxxxxxx | Managed Amazon S3 prefix list com.amazonaws.MyRegion.s3. | | All Traffic | All | All | sg-xxxxxxxxxxxxxxxxx | The ID of the ElasticMapReduce-Core-Private security group. | | All Traffic | All | All | sg-xxxxxxxxxxxxxxxxx | The ID of the ElasticMapReduce-Primary-Private security group. | | Custom TCP | TCP | 9443 | sg-xxxxxxxxxxxxxxxxx | The ID of the ElasticMapReduce-ServiceAccess security group. | ## Amazon EMR-managed security group for service access (private subnets) The default managed security group for service access in private subnets has the **Group Name** of **ElasticMapReduce-ServiceAccess**. It has inbound rules, and outbound rules that allow traffic over HTTPS (port 8443, port 9443) to the other managed security groups in private subnets. These rules allow the cluster manager to communicate with the primary node and with core and task nodes. The same rules are needed if you are using custom security groups. | Type | Protocol | Port range | Source | Details | | --- | --- | --- | --- | --- | | Inbound rules Required for Amazon EMR clusters with Amazon EMR release 5.30.0 and later. | | Custom TCP | TCP | 9443 | The Group ID of the managed security group for primary instance. | This rule allows the communication between primary instance's security group to the service access security group. | | Outbound rules Required for all Amazon EMR clusters | | Custom TCP | TCP | 8443 | The Group ID of the managed security group for primary instance. | These rules allow the cluster manager to communicate with the primary node and with core and task nodes. | | Custom TCP | TCP | 8443 | The Group ID of the managed security group for core and task instances. | These rules allow the cluster manager to communicate with the primary node and with core and task nodes. | # Working with additional security groups for an Amazon EMR cluster Whether you use the default managed security groups or specify custom managed security groups, you can use additional security groups. Additional security groups give you the flexibility to tailor access between different clusters and from external clients, resources, and applications. Consider the following scenario as an example. You have multiple clusters that you need to communicate with each other, but you want to allow inbound SSH access to the primary instance for only a particular subset of clusters. To do this, you can use the same set of managed security groups for the clusters. You then create additional security groups that allow inbound SSH access from trusted clients, and specify the additional security groups for the primary instance to each cluster in the subset. You can apply up to 15 additional security groups for the primary instance, 15 for core and task instances, and 15 for service access (in private subnets). If necessary, you can specify the same additional security group for primary instances, core and task instances, and service access. The maximum number of security groups and rules in your account is subject to account limits. For more information, see [Security group limits](https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html#vpc-limits-security-groups) in the *Amazon VPC User Guide*. # Specifying Amazon EMR-managed and additional security groups You can specify security groups using the AWS Management Console, the AWS CLI, or the Amazon EMR API. If you don't specify security groups, Amazon EMR creates default security groups. Specifying additional security groups is optional. You can assign additional security groups for primary instances, core and task instances, and service access (private subnets only). ------ #### [ Console ] **To specify security groups with the console** 1. Sign in to the AWS Management Console, and open the Amazon EMR console at [https://console.aws.amazon.com/emr](https://console.aws.amazon.com/emr). 1. Under **EMR on EC2** in the left navigation pane, choose **Clusters**, and then choose **Create cluster**. 1. Under **Networking**, select the arrow next to **EC2 security groups (firewall)** to expand this section. Under **Primary node** and **Core and task nodes**, the default Amazon EMR managed security groups are selected by default. If you use a private subnet, you also have the option to select a security group for **Service access**. 1. To change your Amazon EMR managed security group, use the **Choose security groups** dropdown menu to select a different option from the **Amazon EMR-managed security group** list of options. You have one Amazon EMR managed security group for both **Primary node** and **Core and task nodes**. 1. To add custom security groups, use the same **Choose security groups** dropdown menu to select up to four custom security groups from the **Custom security group** list of options. You can have up to four custom security groups for both **Primary node** and **Core and task nodes**. 1. Choose any other options that apply to your cluster. 1. To launch your cluster, choose **Create cluster**. ------ ## Specifying security groups with the AWS CLI To specify security groups using the AWS CLI you use the `create-cluster` command with the following parameters of the `--ec2-attributes` option: | Parameter | Description | | --- | --- | | `EmrManagedPrimarySecurityGroup` | Use this parameter to specify a custom managed security group for the primary instance. If this parameter is specified, `EmrManagedCoreSecurityGroup` must also be specified. For clusters in private subnets, `ServiceAccessSecurityGroup` must also be specified. | | `EmrManagedCoreSecurityGroup` | Use this parameter to specify a custom managed security group for core and task instances. If this parameter is specified, `EmrManagedPrimarySecurityGroup` must also be specified. For clusters in private subnets, `ServiceAccessSecurityGroup` must also be specified. | | `ServiceAccessSecurityGroup` | Use this parameter to specify a custom managed security group for service access, which applies only to clusters in private subnets. The security group you specify as `ServiceAccessSecurityGroup` should not be used for any other purpose and should also be reserved for Amazon EMR. If this parameter is specified, `EmrManagedPrimarySecurityGroup` must also be specified. | | `AdditionalPrimarySecurityGroups` | Use this parameter to specify up to four additional security groups for the primary instance. | | `AdditionalCoreSecurityGroups` | Use this parameter to specify up to four additional security groups for core and task instances. | **Example — specify custom Amazon EMR-managed security groups and additional security groups** The following example specifies custom Amazon EMR managed security groups for a cluster in a private subnet, multiple additional security groups for the primary instance, and a single additional security group for core and task instances. Linux line continuation characters (\$1) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^). ``` 1. aws emr create-cluster --name "ClusterCustomManagedAndAdditionalSGs" \ 2. --release-label emr-emr-7.12.0 --applications Name=Hue Name=Hive \ 3. Name=Pig --use-default-roles --ec2-attributes \ 4. SubnetIds=subnet-xxxxxxxxxxxx,KeyName=myKey,\ 5. ServiceAccessSecurityGroup=sg-xxxxxxxxxxxx,\ 6. EmrManagedPrimarySecurityGroup=sg-xxxxxxxxxxxx,\ 7. EmrManagedCoreSecurityGroup=sg-xxxxxxxxxxx,\ 8. AdditionalPrimarySecurityGroups=['sg-xxxxxxxxxxx',\ 9. 'sg-xxxxxxxxxxx','sg-xxxxxxxxxx'],\ 10. AdditionalCoreSecurityGroups=sg-xxxxxxxxxxx \ 11. --instance-type m5.xlarge ``` For more information, see [create-cluster](https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html) in the *AWS CLI Command Reference*. # Specifying EC2 security groups for EMR Notebooks When you create an EMR notebook, two security groups are used to control network traffic between the EMR notebook and the Amazon EMR cluster when you use the notebook editor. The default security groups have minimal rules that allow only network traffic between the EMR Notebooks service and the clusters to which notebooks are attached. An EMR notebook uses [Apache Livy](https://livy.incubator.apache.org/) to communicate with the cluster via a proxy through TCP Port 18888. When you create custom security groups with rules that you tailor to your environment, you can limit network traffic so that only a subset of notebooks can run code within the notebook editor on particular clusters. The cluster uses your custom security in addition to the default security groups for the cluster. For more information, see [Control network traffic with security groups](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-security-groups.html) in the *Amazon EMR Management Guide* and [Specifying EC2 security groups for EMR Notebooks](#emr-managed-notebooks-security-groups). ## Default EC2 security group for the primary instance The default EC2 security group for the primary instance is associated with the primary instance in addition to the cluster's security groups for the primary instance. Group Name: **ElasticMapReduceEditors-Livy** **Rules** + Inbound Allow TCP Port 18888 from any resources in the default EC2 security group for EMR Notebooks + Outbound None ## Default EC2 security group for EMR Notebooks The default EC2 security group for the EMR notebook is associated with the notebook editor for any EMR notebook to which it is assigned. Group Name: **ElasticMapReduceEditors-Editor** **Rules** + Inbound None + Outbound Allow TCP Port 18888 to any resources in the default EC2 security group for EMR Notebooks. ## Custom EC2 security group for EMR Notebooks when associating Notebooks with Git repositories To link a Git repository to your notebook, the security group for the EMR notebook must include an outbound rule so that the notebook can route traffic to the internet. It is recommended that you create a new security group for this purpose. Updating the default **ElasticMapReduceEditors-Editor** security group may give the same outbound rules to other notebooks that are attached to this security group. **Rules** + Inbound None + Outbound Allow the notebook to route traffic to the internet via the cluster, as the following example demonstrates. The value 0.0.0.0/0 is used for example purposes. You can modify this rule to specify the IP address(es) for your Git-based repositories. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-security-groups.html) # Using Amazon EMR block public access Amazon EMR *block public access (BPA)* prevents you from launching a cluster in a public subnet if the cluster has a security configuration that allows inbound traffic from public IP addresses on a port. **Important** *Block public access* is enabled by default. To increase account protection, we recommend that you keep it enabled. ## Understanding block public access You can use the *block public access* account-level configuration to centrally manage public network access to Amazon EMR clusters. When a user from your AWS account launches a cluster, Amazon EMR checks the port rules in the security group for the cluster and compares them with your inbound traffic rules. If the security group has an inbound rule that opens ports to the public IP addresses IPv4 0.0.0.0/0 or IPv6 ::/0, and those ports aren't specified as exceptions for your account, Amazon EMR doesn't let the user create the cluster. If a user modifies the security group rules for a running cluster in a public subnet to have a public access rule that violates the BPA configuration for your account, Amazon EMR revokes the new rule if it has permission to do so. If Amazon EMR doesn't have permission to revoke the rule, it creates an event in the AWS Health dashboard that describes the violation. To grant the revoke rule permission to Amazon EMR, see [Configure Amazon EMR to revoke security group rules](#revoke-block-public-access). Block public access is enabled by default for all clusters in every AWS Region for your AWS account. BPA applies to the entire lifecycle of a cluster, but doesn't apply to clusters that you create in private subnets. You can configure exceptions to the BPA rule; port 22 is an exception by default. For more information on setting exceptions, see [Configure block public access](#configure-block-public-access). ## Configure block public access You can update security groups and the block public access configuration in your accounts at any time. You can turn block public access (BPA) settings on and off with the AWS Management Console, the AWS Command Line Interface (AWS CLI), and the Amazon EMR API. Settings apply across your account on a Region-by-Region basis. To maintain cluster security, we recommend that you use BPA. ------ #### [ Console ] **To configure block public access with the console** 1. Sign in to the AWS Management Console, then open the Amazon EMR console at [https://console.aws.amazon.com/emr](https://console.aws.amazon.com/emr). 1. On the top navigation bar, select the **Region** that you want to configure if it's not already selected. 1. Under **EMR on EC2** in the left navigation pane, choose **Block public access**. 1. Under **Block public access settings**, complete the following steps. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-block-public-access.html) ------ #### [ AWS CLI ] **To configure block public access using the AWS CLI** + Use the `aws emr put-block-public-access-configuration` command to configure block public access as shown in the following examples. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-block-public-access.html) ------ ## Configure Amazon EMR to revoke security group rules Amazon EMR needs permission to revoke security group rules and comply with your block public access configuration. You can use one of the following approaches to give Amazon EMR the permission that it needs: + **(Recommended)** Attach the `AmazonEMRServicePolicy_v2` managed policy to the service role. For more information, see [Service role for Amazon EMR (EMR role)](emr-iam-role.md). + Create a new inline policy that allows the `ec2:RevokeSecurityGroupIngress` action on security groups. For more information about how to modify a role permissions policy, see **Modifying a role permissions policy** with the [IAM Console](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_permissions-policy), [AWS API](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-api.html#roles-modify_permissions-policy-api), and [AWS CLI](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-cli.html#roles-modify_permissions-policy-cli) in the * IAM User Guide*. ## Resolve block public access violations If a block public access violation occurs, you can mitigate it with one of the following actions: + If you want to access a web interface on your cluster, use one of the options described in [View web interfaces hosted on Amazon EMR clusters](emr-web-interfaces.md) to access the interface through SSH (port 22). + To allow traffic to the cluster from specific IP addresses rather than from the public IP address, add a security group rule. For more information, see [Add rules to a security group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#adding-security-group-rule) in the *Amazon EC2 Getting Started Guide*. + **(Not recommended)** You can configure Amazon EMR BPA exceptions to include the desired port or range of ports. When you specify a BPA exception, you introduce risk with an unprotected port. If you plan to specify an exception, you should remove the exception as soon as it's no longer needed. For more information, see [Configure block public access](#configure-block-public-access). ## Identify clusters associated with security group rules You might need to identify all of the clusters that are associated with a given security group rule, or to find the security group rule for a given cluster. + If you know the security group, then you can identify its associated clusters if you find the network interfaces for the security group. For more information, see [How can I find the resources associated with an Amazon EC2 security group?](https://forums.aws.amazon.com/knowledge-center/ec2-find-security-group-resources) on AWS re:Post. The Amazon EC2 instances that are attached to these network interfaces will be tagged with the ID of the cluster that they belong to. + If you want to find the security groups for a known cluster, follow the steps in [View Amazon EMR cluster status and details](emr-manage-view-clusters.md). You can find the security groups for the cluster in the **Network and security** panel in the console, or in the `Ec2InstanceAttributes` field from the AWS CLI.