**Help improve this page**
To contribute to this user guide, choose the **Edit this page on GitHub** link that is located in the right pane of every page.
# Manage compute resources by using nodes
A Kubernetes node is a machine that runs containerized applications. Each node has the following components:
+ ** [Container runtime](https://kubernetes.io/docs/setup/production-environment/container-runtimes/) ** – Software that’s responsible for running the containers.
+ ** [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) ** – Makes sure that containers are healthy and running within their associated Pod.
+ ** [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) ** – Maintains network rules that allow communication to your Pods.
For more information, see [Nodes](https://kubernetes.io/docs/concepts/architecture/nodes/) in the Kubernetes documentation.
Your Amazon EKS cluster can schedule Pods on any combination of [EKS Auto Mode managed nodes](automode.md), [self-managed nodes](worker.md), [Amazon EKS managed node groups](managed-node-groups.md), [AWS Fargate](fargate.md), and [Amazon EKS Hybrid Nodes](hybrid-nodes-overview.md). To learn more about nodes deployed in your cluster, see [View Kubernetes resources in the AWS Management Console](view-kubernetes-resources.md).
**Note**
Excluding hybrid nodes, nodes must be in the same VPC as the subnets you selected when you created the cluster. However, the nodes don’t have to be in the same subnets.
## Compare compute options
The following table provides several criteria to evaluate when deciding which options best meet your requirements. Self-managed nodes are another option which support all of the criteria listed, but they require a lot more manual maintenance. For more information, see [Maintain nodes yourself with self-managed nodes](worker.md).
**Note**
Bottlerocket has some specific differences from the general information in this table. For more information, see the Bottlerocket [documentation](https://github.com/bottlerocket-os/bottlerocket/blob/develop/README.md) on GitHub.
| Criteria | EKS managed node groups | EKS Auto Mode | Amazon EKS Hybrid Nodes |
| --- | --- | --- | --- |
| Can be deployed to [AWS Outposts](https://docs.aws.amazon.com/outposts/latest/userguide/what-is-outposts.html) | No | No | No |
| Can be deployed to an [AWS Local Zone](local-zones.md) | Yes | No | No |
| Can run containers that require Windows | Yes | No | No |
| Can run containers that require Linux | Yes | Yes | Yes |
| Can run workloads that require the Inferentia chip | [Yes](inferentia-support.md) – Amazon Linux nodes only | Yes | No |
| Can run workloads that require a GPU | [Yes](eks-optimized-ami.md#gpu-ami) – Amazon Linux nodes only | Yes | Yes |
| Can run workloads that require Arm processors | [Yes](eks-optimized-ami.md#arm-ami) | Yes | Yes |
| Can run AWS [Bottlerocket](https://aws.amazon.com/bottlerocket/) | Yes | Yes | No |
| Pods share CPU, memory, storage, and network resources with other Pods. | Yes | Yes | Yes |
| Must deploy and manage Amazon EC2 instances | Yes | No - Learn about [EC2 managed instances](automode-learn-instances.md) | Yes – the on-premises physical or virtual machines are managed by you with your choice of tooling. |
| Must secure, maintain, and patch the operating system of Amazon EC2 instances | Yes | No | Yes – the operating system running on your physical or virtual machines are managed by you with your choice of tooling. |
| Can provide bootstrap arguments at deployment of a node, such as extra [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) arguments. | Yes – Using `eksctl` or a [launch template](launch-templates.md) with a custom AMI. | No - [Use a `NodeClass` to configure nodes](create-node-class.md) | Yes - you can customize bootstrap arguments with nodeadm. See [Hybrid nodes `nodeadm` reference](hybrid-nodes-nodeadm.md). |
| Can assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node. | Yes – Using a launch template with a custom AMI. For more information, see [Customize managed nodes with launch templates](launch-templates.md). | No | Yes - see [Configure CNI for hybrid nodes](hybrid-nodes-cni.md). |
| Can SSH into node | Yes | No - [Learn how to troubleshoot nodes](auto-troubleshoot.md) | Yes |
| Can deploy your own custom AMI to nodes | Yes – Using a [launch template](launch-templates.md) | No | Yes |
| Can deploy your own custom CNI to nodes | Yes – Using a [launch template](launch-templates.md) with a custom AMI | No | Yes |
| Must update node AMI on your own | [Yes](update-managed-node-group.md) – If you deployed an Amazon EKS optimized AMI, you’re notified in the Amazon EKS console when updates are available. You can perform the update with one-click in the console. If you deployed a custom AMI, you’re not notified in the Amazon EKS console when updates are available. You must perform the update on your own. | No | Yes - the operating system running on your physical or virtual machines is managed by you with your choice of tooling. See [Prepare operating system for hybrid nodes](hybrid-nodes-os.md). |
| Must update node Kubernetes version on your own | [Yes](update-managed-node-group.md) – If you deployed an Amazon EKS optimized AMI, you’re notified in the Amazon EKS console when updates are available. You can perform the update with one-click in the console. If you deployed a custom AMI, you’re not notified in the Amazon EKS console when updates are available. You must perform the update on your own. | No | Yes - you manage hybrid nodes upgrades with your own choice of tooling or with `nodeadm`. See [Upgrade hybrid nodes for your cluster](hybrid-nodes-upgrade.md). |
| Can use Amazon EBS storage with Pods | [Yes](ebs-csi.md) | Yes, as an integrated capability. Learn how to [create a storage class.](create-storage-class.md) | No |
| Can use Amazon EFS storage with Pods | [Yes](efs-csi.md) | Yes | No |
| Can use Amazon S3 Files storage with Pods | [Yes](s3files-csi.md) | Yes | No |
| Can use Amazon FSx for Lustre storage with Pods | [Yes](fsx-csi.md) | Yes | No |
| Can use Network Load Balancer for services | [Yes](network-load-balancing.md) | Yes | Yes - must use target type `ip`. |
| Pods can run in a public subnet | Yes | Yes | No - pods run in on-premises environment. |
| Can assign different VPC security groups to individual Pods | [Yes](security-groups-for-pods.md) – Linux nodes only | No | No |
| Can run Kubernetes DaemonSets | Yes | Yes | Yes |
| Support `HostPort` and `HostNetwork` in the Pod manifest | Yes | Yes | Yes |
| AWS Region availability | [All Amazon EKS supported regions](https://docs.aws.amazon.com/general/latest/gr/eks.html) | [All Amazon EKS supported regions](https://docs.aws.amazon.com/general/latest/gr/eks.html) | [All Amazon EKS supported regions](https://docs.aws.amazon.com/general/latest/gr/eks.html) except the AWS GovCloud (US) Regions and the China Regions. |
| Can run containers on Amazon EC2 dedicated hosts | Yes | No | No |
| Pricing | Cost of Amazon EC2 instance that runs multiple Pods. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/). | When EKS Auto Mode is enabled in your cluster, you pay a separate fee, in addition to the standard EC2 instance charges, for the instances launched using Auto Mode’s compute capability. The amount varies with the instance type launched and the AWS region where your cluster is located. For more information, see [Amazon EKS pricing](https://aws.amazon.com/eks/pricing/). | Cost of hybrid nodes vCPU per hour. For more information, see [Amazon EKS pricing](https://aws.amazon.com/eks/pricing/). |
# Simplify node lifecycle with managed node groups
Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters.
With Amazon EKS managed node groups, you don’t need to separately provision or register the Amazon EC2 instances that provide compute capacity to run your Kubernetes applications. You can create, automatically update, or terminate nodes for your cluster with a single operation. Node updates and terminations automatically drain nodes to ensure that your applications stay available.
Every managed node is provisioned as part of an Amazon EC2 Auto Scaling group that’s managed for you by Amazon EKS. Every resource including the instances and Auto Scaling groups runs within your AWS account. Each node group runs across multiple Availability Zones that you define.
Managed node groups can also optionally leverage node auto repair, which continuously monitors the health of nodes. It automatically reacts to detected problems and replaces nodes when possible. This helps overall availability of the cluster with minimal manual intervention. For more information, see [Detect node health issues and enable automatic node repair](node-health.md).
You can add a managed node group to new or existing clusters using the Amazon EKS console, `eksctl`, AWS CLI, AWS API, or infrastructure as code tools including AWS CloudFormation. Nodes launched as part of a managed node group are automatically tagged for auto-discovery by the Kubernetes [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md). You can use the node group to apply Kubernetes labels to nodes and update them at any time.
There are no additional costs to use Amazon EKS managed node groups, you only pay for the AWS resources you provision. These include Amazon EC2 instances, Amazon EBS volumes, Amazon EKS cluster hours, and any other AWS infrastructure. There are no minimum fees and no upfront commitments.
To get started with a new Amazon EKS cluster and managed node group, see [Get started with Amazon EKS – AWS Management Console and AWS CLI](getting-started-console.md).
To add a managed node group to an existing cluster, see [Create a managed node group for your cluster](create-managed-node-group.md).
## Managed node groups concepts
+ Amazon EKS managed node groups create and manage Amazon EC2 instances for you.
+ Every managed node is provisioned as part of an Amazon EC2 Auto Scaling group that’s managed for you by Amazon EKS. Moreover, every resource including Amazon EC2 instances and Auto Scaling groups run within your AWS account.
+ Amazon EKS periodically syncs the managed node group’s scaling configuration to match the actual Auto Scaling group values. If an external actor such as Cluster Autoscaler modifies the Auto Scaling group’s size, `DescribeNodegroup` will eventually reflect those changes. When you initiate a node group update or upgrade without explicitly modifying the scaling configuration, the workflow uses the current Auto Scaling group values rather than the node group’s stored scaling configuration. The stored scaling configuration only takes precedence when you explicitly include it in an `UpdateNodegroupConfig` request.
+ The Auto Scaling group of a managed node group spans every subnet that you specify when you create the group.
+ Amazon EKS tags managed node group resources so that they are configured to use the Kubernetes [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md).
**Important**
If you are running a stateful application across multiple Availability Zones that is backed by Amazon EBS volumes and using the Kubernetes [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), you should configure multiple node groups, each scoped to a single Availability Zone. In addition, you should enable the `--balance-similar-node-groups` feature.
+ You can use a custom launch template for a greater level of flexibility and customization when deploying managed nodes. For example, you can specify extra `kubelet` arguments and use a custom AMI. For more information, see [Customize managed nodes with launch templates](launch-templates.md). If you don’t use a custom launch template when first creating a managed node group, there is an auto-generated launch template. Don’t manually modify this auto-generated template or errors occur.
+ Amazon EKS follows the shared responsibility model for CVEs and security patches on managed node groups. When managed nodes run an Amazon EKS optimized AMI, Amazon EKS is responsible for building patched versions of the AMI when bugs or issues are reported. We can publish a fix. However, you’re responsible for deploying these patched AMI versions to your managed node groups. When managed nodes run a custom AMI, you’re responsible for building patched versions of the AMI when bugs or issues are reported and then deploying the AMI. For more information, see [Update a managed node group for your cluster](update-managed-node-group.md).
+ Amazon EKS managed node groups can be launched in both public and private subnets. If you launch a managed node group in a public subnet on or after April 22, 2020, the subnet must have `MapPublicIpOnLaunch` set to true for the instances to successfully join a cluster. If the public subnet was created using `eksctl` or the [Amazon EKS vended AWS CloudFormation templates](creating-a-vpc.md) on or after March 26, 2020, then this setting is already set to true. If the public subnets were created before March 26, 2020, you must change the setting manually. For more information, see [Modifying the public IPv4 addressing attribute for your subnet](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#subnet-public-ip).
+ When deploying a managed node group in private subnets, you must ensure that it can access Amazon ECR for pulling container images. You can do this by connecting a NAT gateway to the route table of the subnet or by adding the following [AWS PrivateLink VPC endpoints](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html#ecr-setting-up-vpc-create):
+ Amazon ECR API endpoint interface – `com.amazonaws.region-code.ecr.api`
+ Amazon ECR Docker registry API endpoint interface – `com.amazonaws.region-code.ecr.dkr`
+ Amazon S3 gateway endpoint – `com.amazonaws.region-code.s3`
For other commonly-used services and endpoints, see [Deploy private clusters with limited internet access](private-clusters.md).
+ Managed node groups can’t be deployed on [AWS Outposts](eks-outposts.md) or in [AWS Wavelength](https://docs.aws.amazon.com/wavelength/). Managed node groups can be created on [AWS Local Zones](https://aws.amazon.com/about-aws/global-infrastructure/localzones/). For more information, see [Launch low-latency EKS clusters with AWS Local Zones](local-zones.md).
+ You can create multiple managed node groups within a single cluster. For example, you can create one node group with the standard Amazon EKS optimized Amazon Linux AMI for some workloads and another with the GPU variant for workloads that require GPU support.
+ If your managed node group encounters an [Amazon EC2 instance status check](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html) failure, Amazon EKS returns an error code to help you to diagnose the issue. For more information, see [Managed node group error codes](troubleshooting.md#troubleshoot-managed-node-groups).
+ Amazon EKS adds Kubernetes labels to managed node group instances. These Amazon EKS provided labels are prefixed with `eks.amazonaws.com`.
+ Amazon EKS automatically drains nodes using the Kubernetes API during terminations or updates.
+ Pod disruption budgets aren’t respected when terminating a node with `AZRebalance` or reducing the desired node count. These actions try to evict Pods on the node. But if it takes more than 15 minutes, the node is terminated regardless of whether all Pods on the node are terminated. To extend the period until the node is terminated, add a lifecycle hook to the Auto Scaling group. For more information, see [Add lifecycle hooks](https://docs.aws.amazon.com/autoscaling/ec2/userguide/adding-lifecycle-hooks.html) in the *Amazon EC2 Auto Scaling User Guide*.
+ In order to run the drain process correctly after receiving a Spot interruption notification or a capacity rebalance notification, `CapacityRebalance` must be set to `true`.
+ Updating managed node groups respects the Pod disruption budgets that you set for your Pods. For more information, see [Understand each phase of node updates](managed-node-update-behavior.md).
+ There are no additional costs to use Amazon EKS managed node groups. You only pay for the AWS resources that you provision.
+ If you want to encrypt Amazon EBS volumes for your nodes, you can deploy the nodes using a launch template. To deploy managed nodes with encrypted Amazon EBS volumes without using a launch template, encrypt all new Amazon EBS volumes created in your account. For more information, see [Encryption by default](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html#encryption-by-default) in the *Amazon EC2 User Guide*.
## Managed node group capacity types
When creating a managed node group, you can choose either the On-Demand or Spot capacity type. Amazon EKS deploys a managed node group with an Amazon EC2 Auto Scaling group that either contains only On-Demand or only Amazon EC2 Spot Instances. You can schedule Pods for fault tolerant applications to Spot managed node groups, and fault intolerant applications to On-Demand node groups within a single Kubernetes cluster. By default, a managed node group deploys On-Demand Amazon EC2 instances.
### On-Demand
With On-Demand Instances, you pay for compute capacity by the second, with no long-term commitments.
By default, if you don’t specify a **Capacity Type**, the managed node group is provisioned with On-Demand Instances. A managed node group configures an Amazon EC2 Auto Scaling group on your behalf with the following settings applied:
+ The allocation strategy to provision On-Demand capacity is set to `prioritized`. Managed node groups use the order of instance types passed in the API to determine which instance type to use first when fulfilling On-Demand capacity. For example, you might specify three instance types in the following order: `c5.large`, `c4.large`, and `c3.large`. When your On-Demand Instances are launched, the managed node group fulfills On-Demand capacity by starting with `c5.large`, then `c4.large`, and then `c3.large`. For more information, see [Amazon EC2 Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-purchase-options.html#asg-allocation-strategies) in the *Amazon EC2 Auto Scaling User Guide*.
+ Amazon EKS adds the following Kubernetes label to all nodes in your managed node group that specifies the capacity type: `eks.amazonaws.com/capacityType: ON_DEMAND`. You can use this label to schedule stateful or fault intolerant applications on On-Demand nodes.
### Spot
Amazon EC2 Spot Instances are spare Amazon EC2 capacity that offers steep discounts off of On-Demand prices. Amazon EC2 Spot Instances can be interrupted with a two-minute interruption notice when EC2 needs the capacity back. For more information, see [Spot Instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) in the *Amazon EC2 User Guide*. You can configure a managed node group with Amazon EC2 Spot Instances to optimize costs for the compute nodes running in your Amazon EKS cluster.
To use Spot Instances inside a managed node group, create a managed node group by setting the capacity type as `spot`. A managed node group configures an Amazon EC2 Auto Scaling group on your behalf with the following Spot best practices applied:
+ To ensure that your Spot nodes are provisioned in the optimal Spot capacity pools, the allocation strategy is set to one of the following:
+ `price-capacity-optimized` (PCO) – When creating new node groups in a cluster with Kubernetes version `1.28` or higher, the allocation strategy is set to `price-capacity-optimized`. However, the allocation strategy won’t be changed for node groups already created with `capacity-optimized` before Amazon EKS managed node groups started to support PCO.
+ `capacity-optimized` (CO) – When creating new node groups in a cluster with Kubernetes version `1.27` or lower, the allocation strategy is set to `capacity-optimized`.
To increase the number of Spot capacity pools available for allocating capacity from, configure a managed node group to use multiple instance types.
+ Amazon EC2 Spot Capacity Rebalancing is enabled so that Amazon EKS can gracefully drain and rebalance your Spot nodes to minimize application disruption when a Spot node is at elevated risk of interruption. For more information, see [Amazon EC2 Auto Scaling Capacity Rebalancing](https://docs.aws.amazon.com/autoscaling/ec2/userguide/capacity-rebalance.html) in the *Amazon EC2 Auto Scaling User Guide*.
+ When a Spot node receives a rebalance recommendation, Amazon EKS automatically attempts to launch a new replacement Spot node.
+ If a Spot two-minute interruption notice arrives before the replacement Spot node is in a `Ready` state, Amazon EKS starts draining the Spot node that received the rebalance recommendation. Amazon EKS drains the node on a best-effort basis. As a result, there’s no guarantee that Amazon EKS will wait for the replacement node to join the cluster before draining the existing node.
+ When a replacement Spot node is bootstrapped and in the `Ready` state on Kubernetes, Amazon EKS cordons and drains the Spot node that received the rebalance recommendation. Cordoning the Spot node ensures that the service controller doesn’t send any new requests to this Spot node. It also removes it from its list of healthy, active Spot nodes. Draining the Spot node ensures that running Pods are evicted gracefully.
+ Amazon EKS adds the following Kubernetes label to all nodes in your managed node group that specifies the capacity type: `eks.amazonaws.com/capacityType: SPOT`. You can use this label to schedule fault tolerant applications on Spot nodes.
**Important**
EC2 issues a [Spot interruption notice](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html) two minutes prior to terminating your Spot Instance. However, Pods on Spot nodes may not receive the full 2-minute window for graceful shutdown. When EC2 issues the notice, there is a delay before Amazon EKS begins evicting Pods. Evictions occur sequentially to protect the Kubernetes API server, so during multiple simultaneous Spot reclamations, some Pods may receive delayed eviction notices. Pods may be forcibly terminated without receiving termination signals, particularly on nodes with high Pod density, during concurrent reclamations, or when using long termination grace periods. For Spot workloads, we recommend designing applications to be interruption-tolerant, using termination grace periods of 30 seconds or less, avoiding long-running preStop hooks, and monitoring Pod eviction metrics to understand actual grace periods in your clusters. For workloads requiring guaranteed graceful termination, we recommend using On-Demand capacity instead.
When deciding whether to deploy a node group with On-Demand or Spot capacity, you should consider the following conditions:
+ Spot Instances are a good fit for stateless, fault-tolerant, flexible applications. These include batch and machine learning training workloads, big data ETLs such as Apache Spark, queue processing applications, and stateless API endpoints. Because Spot is spare Amazon EC2 capacity, which can change over time, we recommend that you use Spot capacity for interruption-tolerant workloads. More specifically, Spot capacity is suitable for workloads that can tolerate periods where the required capacity isn’t available.
+ We recommend that you use On-Demand for applications that are fault intolerant. This includes cluster management tools such as monitoring and operational tools, deployments that require `StatefulSets`, and stateful applications, such as databases.
+ To maximize the availability of your applications while using Spot Instances, we recommend that you configure a Spot managed node group to use multiple instance types. We recommend applying the following rules when using multiple instance types:
+ Within a managed node group, if you’re using the [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), we recommend using a flexible set of instance types with the same amount of vCPU and memory resources. This is to ensure that the nodes in your cluster scale as expected. For example, if you need four vCPUs and eight GiB memory, use `c3.xlarge`, `c4.xlarge`, `c5.xlarge`, `c5d.xlarge`, `c5a.xlarge`, `c5n.xlarge`, or other similar instance types.
+ To enhance application availability, we recommend deploying multiple Spot managed node groups. For this, each group should use a flexible set of instance types that have the same vCPU and memory resources. For example, if you need 4 vCPUs and 8 GiB memory, we recommend that you create one managed node group with `c3.xlarge`, `c4.xlarge`, `c5.xlarge`, `c5d.xlarge`, `c5a.xlarge`, `c5n.xlarge`, or other similar instance types, and a second managed node group with `m3.xlarge`, `m4.xlarge`, `m5.xlarge`, `m5d.xlarge`, `m5a.xlarge`, `m5n.xlarge` or other similar instance types.
+ When deploying your node group with the Spot capacity type that’s using a custom launch template, use the API to pass multiple instance types. Don’t pass a single instance type through the launch template. For more information about deploying a node group using a launch template, see [Customize managed nodes with launch templates](launch-templates.md).
# Create a managed node group for your cluster
This topic describes how you can launch Amazon EKS managed node groups of nodes that register with your Amazon EKS cluster. After the nodes join the cluster, you can deploy Kubernetes applications to them.
If this is your first time launching an Amazon EKS managed node group, we recommend that you instead follow one of our guides in [Get started with Amazon EKS](getting-started.md). These guides provide walkthroughs for creating an Amazon EKS cluster with nodes.
**Important**
Amazon EKS nodes are standard Amazon EC2 instances. You’re billed based on the normal Amazon EC2 prices. For more information, see [Amazon EC2 Pricing](https://aws.amazon.com/ec2/pricing/).
You can’t create managed nodes in an AWS Region where you have AWS Outposts or AWS Wavelength enabled. You can create self-managed nodes instead. For more information, see [Create self-managed Amazon Linux nodes](launch-workers.md), [Create self-managed Microsoft Windows nodes](launch-windows-workers.md), and [Create self-managed Bottlerocket nodes](launch-node-bottlerocket.md). You can also create a self-managed Amazon Linux node group on an Outpost. For more information, see [Create Amazon Linux nodes on AWS Outposts](eks-outposts-self-managed-nodes.md).
If you don’t [specify an AMI ID](launch-templates.md#launch-template-custom-ami) for the `bootstrap.sh` file included with Amazon EKS optimized Linux or Bottlerocket, managed node groups enforce a maximum number on the value of `maxPods`. For instances with less than 30 vCPUs, the maximum number is `110`. For instances with greater than 30 vCPUs, the maximum number jumps to `250`. This enforcement overrides other `maxPods` configurations, including `maxPodsExpression`. For more information about how `maxPods` is determined and how to customize it, see [How maxPods is determined](choosing-instance-type.md#max-pods-precedence).
+ An existing Amazon EKS cluster. To deploy one, see [Create an Amazon EKS cluster](create-cluster.md).
+ An existing IAM role for the nodes to use. To create one, see [Amazon EKS node IAM role](create-node-role.md). If this role doesn’t have either of the policies for the VPC CNI, the separate role that follows is required for the VPC CNI pods.
+ (Optional, but recommended) The Amazon VPC CNI plugin for Kubernetes add-on configured with its own IAM role that has the necessary IAM policy attached to it. For more information, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md).
+ Familiarity with the considerations listed in [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md). Depending on the instance type you choose, there may be additional prerequisites for your cluster and VPC.
+ To add a Windows managed node group, you must first enable Windows support for your cluster. For more information, see [Deploy Windows nodes on EKS clusters](windows-support.md).
You can create a managed node group with either of the following:
+ [`eksctl`](#eksctl_create_managed_nodegroup)
+ [AWS Management Console](#console_create_managed_nodegroup)
## `eksctl`
**Create a managed node group with eksctl**
This procedure requires `eksctl` version `0.215.0` or later. You can check your version with the following command:
```
eksctl version
```
For instructions on how to install or upgrade `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
1. (Optional) If the **AmazonEKS\$1CNI\$1Policy** managed IAM policy is attached to your [Amazon EKS node IAM role](create-node-role.md), we recommend assigning it to an IAM role that you associate to the Kubernetes `aws-node` service account instead. For more information, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md).
1. Create a managed node group with or without using a custom launch template. Manually specifying a launch template allows for greater customization of a node group. For example, it can allow deploying a custom AMI or providing arguments to the `boostrap.sh` script in an Amazon EKS optimized AMI. For a complete list of every available option and default, enter the following command.
```
eksctl create nodegroup --help
```
In the following command, replace *my-cluster* with the name of your cluster and replace *my-mng* with the name of your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
**Important**
If you don’t use a custom launch template when first creating a managed node group, don’t use one at a later time for the node group. If you didn’t specify a custom launch template, the system auto-generates a launch template that we don’t recommend that you modify manually. Manually modifying this auto-generated launch template might cause errors.
**Without a launch template**
`eksctl` creates a default Amazon EC2 launch template in your account and deploys the node group using a launch template that it creates based on options that you specify. Before specifying a value for `--node-type`, see [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md).
Replace *ami-family* with an allowed keyword. For more information, see [Setting the node AMI Family](https://eksctl.io/usage/custom-ami-support/#setting-the-node-ami-family) in the `eksctl` documentation. Replace *my-key* with the name of your Amazon EC2 key pair or public key. This key is used to SSH into your nodes after they launch.
**Note**
For Windows, this command doesn’t enable SSH. Instead, it associates your Amazon EC2 key pair with the instance and allows you to RDP into the instance.
If you don’t already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For Linux information, see [Amazon EC2 key pairs and Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. For Windows information, see [Amazon EC2 key pairs and Windows instances](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*.
We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
If you want to block Pod access to IMDS, then add the `--disable-pod-imds` option to the following command.
```
eksctl create nodegroup \
--cluster my-cluster \
--region region-code \
--name my-mng \
--node-ami-family ami-family \
--node-type m5.large \
--nodes 3 \
--nodes-min 2 \
--nodes-max 4 \
--ssh-access \
--ssh-public-key my-key
```
Your instances can optionally assign a significantly higher number of IP addresses to Pods, assign IP addresses to Pods from a different CIDR block than the instance’s, and be deployed to a cluster without internet access. For more information, see [Assign more IP addresses to Amazon EKS nodes with prefixes](cni-increase-ip-addresses.md), [Deploy Pods in alternate subnets with custom networking](cni-custom-network.md), and [Deploy private clusters with limited internet access](private-clusters.md) for additional options to add to the previous command.
Managed node groups calculates and applies a single value for the maximum number of Pods that can run on each node of your node group, based on instance type. If you create a node group with different instance types, the smallest value calculated across all instance types is applied as the maximum number of Pods that can run on every instance type in the node group. For more information about how this value is calculated, see [How maxPods is determined](choosing-instance-type.md#max-pods-precedence).
**With a launch template**
The launch template must already exist and must meet the requirements specified in [Launch template configuration basics](launch-templates.md#launch-template-basics). We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
If you want to block Pod access to IMDS, then specify the necessary settings in the launch template.
1. Copy the following contents to your device. Replace the example values and then run the modified command to create the `eks-nodegroup.yaml` file. Several settings that you specify when deploying without a launch template are moved into the launch template. If you don’t specify a `version`, the template’s default version is used.
```
cat >eks-nodegroup.yaml <
**Create a managed node group using the AWS Management Console **
1. Wait for your cluster status to show as `ACTIVE`. You can’t create a managed node group for a cluster that isn’t already `ACTIVE`.
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the name of the cluster that you want to create a managed node group in.
1. Select the **Compute** tab.
1. Choose **Add node group**.
1. On the **Configure node group** page, fill out the parameters accordingly, and then choose **Next**.
+ **Name** – Enter a unique name for your managed node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
+ **Node IAM role** – Choose the node instance role to use with your node group. For more information, see [Amazon EKS node IAM role](create-node-role.md).
**Important**
You can’t use the same role that is used to create any clusters.
We recommend using a role that’s not currently in use by any self-managed node group. Otherwise, you plan to use with a new self-managed node group. For more information, see [Delete a managed node group from your cluster](delete-managed-node-group.md).
+ **Use launch template** – (Optional) Choose if you want to use an existing launch template. Select a **Launch Template Name**. Then, select a **Launch template version**. If you don’t select a version, then Amazon EKS uses the template’s default version. Launch templates allow for more customization of your node group, such as allowing you to deploy a custom AMI, assign a significantly higher number of IP addresses to Pods, assign IP addresses to Pods from a different CIDR block than the instance’s, and deploying nodes to a cluster without outbound internet access. For more information, see [Assign more IP addresses to Amazon EKS nodes with prefixes](cni-increase-ip-addresses.md), [Deploy Pods in alternate subnets with custom networking](cni-custom-network.md), and [Deploy private clusters with limited internet access](private-clusters.md).
The launch template must meet the requirements in [Customize managed nodes with launch templates](launch-templates.md). If you don’t use your own launch template, the Amazon EKS API creates a default Amazon EC2 launch template in your account and deploys the node group using the default launch template.
If you implement [IAM roles for service accounts](iam-roles-for-service-accounts.md), assign necessary permissions directly to every Pod that requires access to AWS services, and no Pods in your cluster require access to IMDS for other reasons, such as retrieving the current AWS Region, then you can also disable access to IMDS for Pods that don’t use host networking in a launch template. For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
+ **Kubernetes labels** – (Optional) You can choose to apply Kubernetes labels to the nodes in your managed node group.
+ **Kubernetes taints** – (Optional) You can choose to apply Kubernetes taints to the nodes in your managed node group. The available options in the **Effect** menu are ` NoSchedule `, ` NoExecute `, and ` PreferNoSchedule `. For more information, see [Recipe: Prevent pods from being scheduled on specific nodes](node-taints-managed-node-groups.md).
+ **Tags** – (Optional) You can choose to tag your Amazon EKS managed node group. These tags don’t propagate to other resources in the node group, such as Auto Scaling groups or instances. For more information, see [Organize Amazon EKS resources with tags](eks-using-tags.md).
1. On the **Set compute and scaling configuration** page, fill out the parameters accordingly, and then choose **Next**.
+ **AMI type** – Select an AMI type. If you are deploying Arm instances, be sure to review the considerations in [Amazon EKS optimized Arm Amazon Linux AMIs](eks-optimized-ami.md#arm-ami) before deploying.
If you specified a launch template on the previous page, and specified an AMI in the launch template, then you can’t select a value. The value from the template is displayed. The AMI specified in the template must meet the requirements in [Specifying an AMI](launch-templates.md#launch-template-custom-ami).
+ **Capacity type** – Select a capacity type. For more information about choosing a capacity type, see [Managed node group capacity types](managed-node-groups.md#managed-node-group-capacity-types). You can’t mix different capacity types within the same node group. If you want to use both capacity types, create separate node groups, each with their own capacity and instance types. See [Reserve GPUs for managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/capacity-blocks-mng.html) for information on provisioning and scaling GPU-accelerated worker nodes.
+ **Instance types** – By default, one or more instance type is specified. To remove a default instance type, select the `X` on the right side of the instance type. Choose the instance types to use in your managed node group. For more information, see [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md).
The console displays a set of commonly used instance types. If you need to create a managed node group with an instance type that’s not displayed, then use `eksctl`, the AWS CLI, AWS CloudFormation, or an SDK to create the node group. If you specified a launch template on the previous page, then you can’t select a value because the instance type must be specified in the launch template. The value from the launch template is displayed. If you selected **Spot** for **Capacity type**, then we recommend specifying multiple instance types to enhance availability.
+ **Disk size** – Enter the disk size (in GiB) to use for your node’s root volume.
If you specified a launch template on the previous page, then you can’t select a value because it must be specified in the launch template.
+ **Desired size** – Specify the current number of nodes that the managed node group should maintain at launch.
**Note**
Amazon EKS doesn’t automatically scale your node group in or out. However, you can configure the Kubernetes Cluster Autoscaler to do this for you. For more information, see [Cluster Autoscaler on AWS](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md).
+ **Minimum size** – Specify the minimum number of nodes that the managed node group can scale in to.
+ **Maximum size** – Specify the maximum number of nodes that the managed node group can scale out to.
+ **Node group update configuration** – (Optional) You can select the number or percentage of nodes to be updated in parallel. These nodes will be unavailable during the update. For **Maximum unavailable**, select one of the following options and specify a **Value**:
+ **Number** – Select and specify the number of nodes in your node group that can be updated in parallel.
+ **Percentage** – Select and specify the percentage of nodes in your node group that can be updated in parallel. This is useful if you have a large number of nodes in your node group.
+ **Node auto repair configuration** – (Optional) If you activate the **Enable node auto repair** checkbox, Amazon EKS will automatically replace nodes when detected issues occur. For more information, see [Detect node health issues and enable automatic node repair](node-health.md).
+ **Warm pool configuration** – (Optional) If you activate the **Enable warm pool configuration** checkbox, Amazon EKS will create warm pools on the ASG. For more information, see [Decrease latency for applications with long boot times using warm pools with managed node groups](warm-pools-managed-node-groups.md).
1. On the **Specify networking** page, fill out the parameters accordingly, and then choose **Next**.
+ **Subnets** – Choose the subnets to launch your managed nodes into.
**Important**
If you are running a stateful application across multiple Availability Zones that is backed by Amazon EBS volumes and using the Kubernetes [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), you should configure multiple node groups, each scoped to a single Availability Zone. In addition, you should enable the `--balance-similar-node-groups` feature.
**Important**
If you choose a public subnet, and your cluster has only the public API server endpoint enabled, then the subnet must have `MapPublicIPOnLaunch` set to `true` for the instances to successfully join a cluster. If the subnet was created using `eksctl` or the [Amazon EKS vended AWS CloudFormation templates](creating-a-vpc.md) on or after March 26, 2020, then this setting is already set to `true`. If the subnets were created with `eksctl` or the AWS CloudFormation templates before March 26, 2020, then you need to change the setting manually. For more information, see [Modifying the public IPv4 addressing attribute for your subnet](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#subnet-public-ip).
If you use a launch template and specify multiple network interfaces, Amazon EC2 won’t auto-assign a public `IPv4` address, even if `MapPublicIpOnLaunch` is set to `true`. For nodes to join the cluster in this scenario, you must either enable the cluster’s private API server endpoint, or launch nodes in a private subnet with outbound internet access provided through an alternative method, such as a NAT Gateway. For more information, see [Amazon EC2 instance IP addressing](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html) in the *Amazon EC2 User Guide*.
+ **Configure SSH access to nodes** (Optional). Enabling SSH allows you to connect to your instances and gather diagnostic information if there are issues. We highly recommend enabling remote access when you create a node group. You can’t enable remote access after the node group is created.
If you chose to use a launch template, then this option isn’t shown. To enable remote access to your nodes, specify a key pair in the launch template and ensure that the proper port is open to the nodes in the security groups that you specify in the launch template. For more information, see [Using custom security groups](launch-templates.md#launch-template-security-groups).
**Note**
For Windows, this command doesn’t enable SSH. Instead, it associates your Amazon EC2 key pair with the instance and allows you to RDP into the instance.
+ For **SSH key pair** (Optional), choose an Amazon EC2 SSH key to use. For Linux information, see [Amazon EC2 key pairs and Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. For Windows information, see [Amazon EC2 key pairs and Windows instances](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. If you chose to use a launch template, then you can’t select one. When an Amazon EC2 SSH key is provided for node groups using Bottlerocket AMIs, the administrative container is also enabled. For more information, see [Admin container](https://github.com/bottlerocket-os/bottlerocket#admin-container) on GitHub.
+ For **Allow SSH remote access from**, if you want to limit access to specific instances, then select the security groups that are associated to those instances. If you don’t select specific security groups, then SSH access is allowed from anywhere on the internet (`0.0.0.0/0`).
1. On the **Review and create** page, review your managed node group configuration and choose **Create**.
If nodes fail to join the cluster, then see [Nodes fail to join cluster](troubleshooting.md#worker-node-fail) in the Troubleshooting chapter.
1. Watch the status of your nodes and wait for them to reach the `Ready` status.
```
kubectl get nodes --watch
```
1. (GPU nodes only) If you chose a GPU instance type and an Amazon EKS optimized accelerated AMI, then you must apply the [NVIDIA device plugin for Kubernetes](https://github.com/NVIDIA/k8s-device-plugin) as a DaemonSet on your cluster. Replace *vX.X.X* with your desired [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin/releases) version before running the following command.
```
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/vX.X.X/deployments/static/nvidia-device-plugin.yml
```
## Install Kubernetes add-ons
Now that you have a working Amazon EKS cluster with nodes, you’re ready to start installing Kubernetes add-ons and deploying applications to your cluster. The following documentation topics help you to extend the functionality of your cluster.
+ The [IAM principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-principal) that created the cluster is the only principal that can make calls to the Kubernetes API server with `kubectl` or the AWS Management Console. If you want other IAM principals to have access to your cluster, then you need to add them. For more information, see [Grant IAM users and roles access to Kubernetes APIs](grant-k8s-access.md) and [Required permissions](view-kubernetes-resources.md#view-kubernetes-resources-permissions).
+ We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
+ Configure the Kubernetes [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md) to automatically adjust the number of nodes in your node groups.
+ Deploy a [sample application](sample-deployment.md) to your cluster.
+ [Organize and monitor cluster resources](eks-managing.md) with important tools for managing your cluster.
# Decrease latency for applications with long boot times using warm pools with managed node groups
When your applications have long initialization or boot times, scale-out events can cause delays—new nodes must fully boot and join the cluster before Pods can be scheduled on them. This latency can impact application availability during traffic spikes or rapid scaling events. Warm pools solve this problem by maintaining a pool of pre-initialized EC2 instances that have already completed the bootup process. During a scale-out event, instances move from the warm pool directly to your cluster, bypassing the time-consuming initialization steps and significantly reducing the time it takes for new capacity to become available. For more information, see [Decrease latency for applications that have long boot times using warm pools](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html) in the *Amazon EC2 Auto Scaling User Guide*.
Amazon EKS managed node groups support Amazon EC2 Auto Scaling warm pools. A warm pool maintains pre-initialized EC2 instances alongside your Auto Scaling group that can quickly join your cluster during scale-out events. Instances in the warm pool have already completed the bootup initialization process and can be kept in a `Stopped`, `Running`, or `Hibernated` state.
Amazon EKS manages warm pools throughout the node group lifecycle using the `AWSServiceRoleForAmazonEKSNodegroup` service-linked role to create, update, and delete warm pool resources.
## How it works
When you configure a warm pool, Amazon EKS creates an EC2 Auto Scaling warm pool attached to your node group’s Auto Scaling group. Instances launch into the warm pool, complete the bootup initialization process, and remain in the configured state (`Running`, `Stopped`, or `Hibernated`) until needed. During scale-out events, instances move from the warm pool to the Auto Scaling group, complete the Amazon EKS initialization process to join the cluster, and become available for pod scheduling. With instance reuse enabled, instances can return to the warm pool during scale-in events.
**Important**
Always configure warm pools through the Amazon EKS API using `create-nodegroup` or `update-nodegroup-config`. Don’t manually modify warm pool settings using the EC2 Auto Scaling API, as this can cause conflicts with Amazon EKS management of the resources.
## Considerations
**Important**
Before configuring warm pools, review the prerequisites and limitations in [Warm pools for Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html) in the *Amazon EC2 Auto Scaling User Guide*. Not all instance types, AMIs, or configurations are supported.
+ **IAM permissions** – The `AWSServiceRoleForAmazonEKSNodegroup` service-linked role (created automatically with your first managed node group) includes the necessary warm pool management permissions.
+ **AMI limitations** – Warm pools don’t support custom AMIs. You must use Amazon EKS optimized AMIs.
+ **Bottlerocket limitations** – If using Bottlerocket AMIs, the `Hibernated` pool state isn’t supported. Use `Stopped` or `Running` pool states only. Additionally, the `reuseOnScaleIn` feature isn’t supported with Bottlerocket AMIs.
+ **Hibernation support** – The `Hibernated` pool state is only supported on specific instance types. See [Hibernation prerequisites](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/hibernating-prerequisites.html) in the *Amazon EC2 User Guide* for supported instance types.
+ **Cost impact** – Creating a warm pool when it’s not required can lead to unnecessary costs.
+ **Capacity planning** – Size your warm pool based on scaling patterns to balance cost and availability. Start with 10-20% of expected peak capacity.
+ **VPC networking** – Ensure sufficient IP addresses for both Auto Scaling group and warm pool instances.
## Configure warm pools
You can configure warm pools when creating a new managed node group or update an existing managed node group to add warm pool support.
### Configuration parameters
+ **enabled** – (boolean) Indicates your intent to attach a warm pool to the managed node group. Required to enable warm pool support.
+ **maxGroupPreparedCapacity** – (integer) Maximum total instances across warm pool and Auto Scaling group combined.
+ **minSize** – (integer) Minimum number of instances to maintain in the warm pool. Default: `0`.
+ **poolState** – (string) State for warm pool instances. Default: `Stopped`.
+ **reuseOnScaleIn** – (boolean) Whether instances return to the warm pool during scale-in events instead of terminating them. Default: `false`. Not supported with Bottlerocket AMIs.
### Using the AWS CLI
You can configure a warm pool when creating a managed node group or add one to an existing node group.
**Create a node group with a warm pool**
```
aws eks create-nodegroup \
--cluster-name my-cluster \
--nodegroup-name my-nodegroup \
--node-role arn:aws:iam::111122223333:role/AmazonEKSNodeRole \
--subnets subnet-12345678 subnet-87654321 \
--region us-east-1 \
--scaling-config minSize=2,maxSize=10,desiredSize=3 \
--warm-pool-config enabled=true,maxGroupPreparedCapacity=8,minSize=2,poolState=Stopped,reuseOnScaleIn=true
```
**Add a warm pool to an existing node group**
```
aws eks update-nodegroup-config \
--cluster-name my-cluster \
--nodegroup-name my-nodegroup \
--region us-east-1 \
--warm-pool-config enabled=true,maxGroupPreparedCapacity=8,minSize=2,poolState=Stopped,reuseOnScaleIn=true
```
## Update configuration
Update warm pool settings at any time using `update-nodegroup-config`. Existing warm pool instances aren’t immediately affected; new settings apply to instances entering the warm pool after the update.
```
aws eks update-nodegroup-config \
--cluster-name my-cluster \
--nodegroup-name my-nodegroup \
--region us-east-1 \
--warm-pool-config enabled=true,maxGroupPreparedCapacity=10,minSize=3,poolState=Running,reuseOnScaleIn=true
```
To disable the warm pool attached to your nodegroup, set `enabled=false`:
```
aws eks update-nodegroup-config \
--cluster-name my-cluster \
--nodegroup-name my-nodegroup \
--region us-east-1 \
--warm-pool-config enabled=false
```
## Additional resources
+ [Warm pools for Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html) in the *Amazon EC2 Auto Scaling User Guide*
+ [Simplify node lifecycle with managed node groups](managed-node-groups.md)
# Update a managed node group for your cluster
When you initiate a managed node group update, Amazon EKS automatically updates your nodes for you, completing the steps listed in [Understand each phase of node updates](managed-node-update-behavior.md). If you’re using an Amazon EKS optimized AMI, Amazon EKS automatically applies the latest security patches and operating system updates to your nodes as part of the latest AMI release version.
There are several scenarios where it’s useful to update your Amazon EKS managed node group’s version or configuration:
+ You have updated the Kubernetes version for your Amazon EKS cluster and want to update your nodes to use the same Kubernetes version.
+ A new AMI release version is available for your managed node group. For more information about AMI versions, see these sections:
+ [Retrieve Amazon Linux AMI version information](eks-linux-ami-versions.md)
+ [Create nodes with optimized Bottlerocket AMIs](eks-optimized-ami-bottlerocket.md)
+ [Retrieve Windows AMI version information](eks-ami-versions-windows.md)
+ You want to adjust the minimum, maximum, or desired count of the instances in your managed node group.
+ You want to add or remove Kubernetes labels from the instances in your managed node group.
+ You want to add or remove AWS tags from your managed node group.
+ You need to deploy a new version of a launch template with configuration changes, such as an updated custom AMI.
+ You have deployed version `1.9.0` or later of the Amazon VPC CNI add-on, enabled the add-on for prefix delegation, and want new AWS Nitro System instances in a node group to support a significantly increased number of Pods. For more information, see [Assign more IP addresses to Amazon EKS nodes with prefixes](cni-increase-ip-addresses.md).
+ You have enabled IP prefix delegation for Windows nodes and want new AWS Nitro System instances in a node group to support a significantly increased number of Pods. For more information, see [Assign more IP addresses to Amazon EKS nodes with prefixes](cni-increase-ip-addresses.md).
If there’s a newer AMI release version for your managed node group’s Kubernetes version, you can update your node group’s version to use the newer AMI version. Similarly, if your cluster is running a Kubernetes version that’s newer than your node group, you can update the node group to use the latest AMI release version to match your cluster’s Kubernetes version.
When a node in a managed node group is terminated due to a scaling operation or update, the Pods in that node are drained first. For more information, see [Understand each phase of node updates](managed-node-update-behavior.md).
## Update a node group version
You can update a node group version with either of the following:
+ [`eksctl`](#eksctl_update_managed_nodegroup)
+ [AWS Management Console](#console_update_managed_nodegroup)
The version that you update to can’t be greater than the control plane’s version.
## `eksctl`
**Update a managed node group using `eksctl` **
Update a managed node group to the latest AMI release of the same Kubernetes version that’s currently deployed on the nodes with the following command. Replace every *example value* with your own values.
```
eksctl upgrade nodegroup \
--name=node-group-name \
--cluster=my-cluster \
--region=region-code
```
**Note**
If you’re upgrading a node group that’s deployed with a launch template to a new launch template version, add `--launch-template-version version-number ` to the preceding command. The launch template must meet the requirements described in [Customize managed nodes with launch templates](launch-templates.md). If the launch template includes a custom AMI, the AMI must meet the requirements in [Specifying an AMI](launch-templates.md#launch-template-custom-ami). When you upgrade your node group to a newer version of your launch template, every node is recycled to match the new configuration of the launch template version that’s specified.
You can’t directly upgrade a node group that’s deployed without a launch template to a new launch template version. Instead, you must deploy a new node group using the launch template to update the node group to a new launch template version.
You can upgrade a node group to the same version as the control plane’s Kubernetes version. For example, if you have a cluster running Kubernetes `1.35`, you can upgrade nodes currently running Kubernetes `1.34` to version `1.35` with the following command.
```
eksctl upgrade nodegroup \
--name=node-group-name \
--cluster=my-cluster \
--region=region-code \
--kubernetes-version=1.35
```
## AWS Management Console
**Update a managed node group using the AWS Management Console **
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the cluster that contains the node group to update.
1. If at least one node group has an available update, a box appears at the top of the page notifying you of the available update. If you select the **Compute** tab, you’ll see **Update now** in the **AMI release version** column in the **Node groups** table for the node group that has an available update. To update the node group, choose **Update now**.
You won’t see a notification for node groups that were deployed with a custom AMI. If your nodes are deployed with a custom AMI, complete the following steps to deploy a new updated custom AMI.
1. Create a new version of your AMI.
1. Create a new launch template version with the new AMI ID.
1. Upgrade the nodes to the new version of the launch template.
1. On the **Update node group version** dialog box, activate or deactivate the following options:
+ **Update node group version** – This option is unavailable if you deployed a custom AMI or your Amazon EKS optimized AMI is currently on the latest version for your cluster.
+ **Change launch template version** – This option is unavailable if the node group is deployed without a custom launch template. You can only update the launch template version for a node group that has been deployed with a custom launch template. Select the **Launch template version** that you want to update the node group to. If your node group is configured with a custom AMI, then the version that you select must also specify an AMI. When you upgrade to a newer version of your launch template, every node is recycled to match the new configuration of the launch template version specified.
1. For **Update strategy**, select one of the following options:
+ **Rolling update** – This option respects the Pod disruption budgets for your cluster. Updates fail if there’s a Pod disruption budget issue that causes Amazon EKS to be unable to gracefully drain the Pods that are running on this node group.
+ **Force update** – This option doesn’t respect Pod disruption budgets. Updates occur regardless of Pod disruption budget issues by forcing node restarts to occur.
1. Choose **Update**.
## Edit a node group configuration
You can modify some of the configurations of a managed node group.
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the cluster that contains the node group to edit.
1. Select the **Compute** tab.
1. Select the node group to edit, and then choose **Edit**.
1. (Optional) On the **Edit node group** page, do the following:
1. Edit the **Node group scaling configuration**.
+ **Desired size** – Specify the current number of nodes that the managed node group should maintain.
+ **Minimum size** – Specify the minimum number of nodes that the managed node group can scale in to.
+ **Maximum size** – Specify the maximum number of nodes that the managed node group can scale out to. For the maximum number of nodes supported in a node group, see [View and manage Amazon EKS and Fargate service quotas](service-quotas.md).
1. (Optional) Add or remove **Kubernetes labels** to the nodes in your node group. The labels shown here are only the labels that you have applied with Amazon EKS. Other labels may exist on your nodes that aren’t shown here.
1. (Optional) Add or remove **Kubernetes taints** to the nodes in your node group. Added taints can have the effect of either ` NoSchedule `, ` NoExecute `, or ` PreferNoSchedule `. For more information, see [Recipe: Prevent pods from being scheduled on specific nodes](node-taints-managed-node-groups.md).
1. (Optional) Add or remove **Tags** from your node group resource. These tags are only applied to the Amazon EKS node group. They don’t propagate to other resources, such as subnets or Amazon EC2 instances in the node group.
1. (Optional) Edit the **Node Group update configuration**. Select either **Number** or **Percentage**.
+ **Number** – Select and specify the number of nodes in your node group that can be updated in parallel. These nodes will be unavailable during update.
+ **Percentage** – Select and specify the percentage of nodes in your node group that can be updated in parallel. These nodes will be unavailable during update. This is useful if you have many nodes in your node group.
1. When you’re finished editing, choose **Save changes**.
**Important**
When updating the node group configuration, modifying the [https://docs.aws.amazon.com/eks/latest/APIReference/API_NodegroupScalingConfig.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_NodegroupScalingConfig.html) does not respect Pod disruption budgets (PDBs). Unlike the [update node group](managed-node-update-behavior.md) process (which drains nodes and respects PDBs during the upgrade phase), updating the scaling configuration causes nodes to be terminated immediately through an Auto Scaling Group (ASG) scale-down call. This happens without considering PDBs, regardless of the target size you’re scaling down to. That means when you reduce the `desiredSize` of an Amazon EKS managed node group, Pods are evicted as soon as the nodes are terminated, without honoring any PDBs.
# Understand each phase of node updates
The Amazon EKS managed worker node upgrade strategy has four different phases described in the following sections.
## Setup phase
The setup phase has these steps:
1. It creates a new Amazon EC2 launch template version for the Auto Scaling Group that’s associated with your node group. The new launch template version uses the target AMI or a custom launch template version for the update.
1. It updates the Auto Scaling Group to use the latest launch template version.
1. It determines the maximum quantity of nodes to upgrade in parallel using the `updateConfig` property for the node group. The maximum unavailable has a quota of 100 nodes. The default value is one node. For more information, see the [updateConfig](https://docs.aws.amazon.com/eks/latest/APIReference/API_UpdateNodegroupConfig.html#API_UpdateNodegroupConfig_RequestSyntax) property in the *Amazon EKS API Reference*.
## Scale up phase
When upgrading the nodes in a managed node group, the upgraded nodes are launched in the same Availability Zone as those that are being upgraded. To guarantee this placement, we use Amazon EC2’s Availability Zone Rebalancing. For more information, see [Availability Zone Rebalancing](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#AutoScalingBehavior.InstanceUsage) in the *Amazon EC2 Auto Scaling User Guide*. To meet this requirement, it’s possible that we’d launch up to two instances per Availability Zone in your managed node group.
The scale up phase has these steps:
1. It increments the Auto Scaling Group’s maximum size and desired size by the larger of either:
+ Up to twice the number of Availability Zones that the Auto Scaling Group is deployed in.
+ The maximum unavailable of upgrade.
For example, if your node group has five Availability Zones and `maxUnavailable` as one, the upgrade process can launch a maximum of 10 nodes. However when `maxUnavailable` is 20 (or anything higher than 10), the process would launch 20 new nodes.
1. After scaling the Auto Scaling Group, it checks if the nodes using the latest configuration are present in the node group. This step succeeds only when it meets these criteria:
+ At least one new node is launched in every Availability Zone where the node exists.
+ Every new node should be in `Ready` state.
+ New nodes should have Amazon EKS applied labels.
These are the Amazon EKS applied labels on the worker nodes in a regular node group:
+ `eks.amazonaws.com/nodegroup-image=$amiName`
+ `eks.amazonaws.com/nodegroup=$nodeGroupName`
These are the Amazon EKS applied labels on the worker nodes in a custom launch template or AMI node group:
+ `eks.amazonaws.com/nodegroup-image=$amiName`
+ `eks.amazonaws.com/nodegroup=$nodeGroupName`
+ `eks.amazonaws.com/sourceLaunchTemplateId=$launchTemplateId`
+ `eks.amazonaws.com/sourceLaunchTemplateVersion=$launchTemplateVersion`
**Note**
When an update or upgrade is initiated without changes to the scaling configuration, the workflow uses the live Auto Scaling group values as the starting point, not the node group’s stored scaling configuration. For more information, see [Managed node groups concepts](managed-node-groups.md#managed-node-group-concepts).
1. It marks nodes as unschedulable to avoid scheduling new Pods. It also labels nodes with `node.kubernetes.io/exclude-from-external-load-balancers=true` to remove the old nodes from load balancers before terminating the nodes.
The following are known reasons which lead to a `NodeCreationFailure` error in this phase:
**Insufficient capacity in the Availability Zone**
There is a possibility that the Availability Zone might not have capacity of requested instance types. It’s recommended to configure multiple instance types while creating a managed node group.
**EC2 instance limits in your account**
You may need to increase the number of Amazon EC2 instances your account can run simultaneously using Service Quotas. For more information, see [EC2 Service Quotas](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html) in the *Amazon Elastic Compute Cloud User Guide for Linux Instances*.
**Custom user data**
Custom user data can sometimes break the bootstrap process. This scenario can lead to the `kubelet` not starting on the node or nodes not getting expected Amazon EKS labels on them. For more information, see [Specifying an AMI](launch-templates.md#launch-template-custom-ami).
**Any changes which make a node unhealthy or not ready**
Node disk pressure, memory pressure, and similar conditions can lead to a node not going to `Ready` state.
**Each node most bootstrap within 15 minutes**
If any node takes more than 15 minutes to bootstrap and join the cluster, it will cause the upgrade to time out. This is the total runtime for bootstrapping a new node measured from when a new node is required to when it joins the cluster. When upgrading a managed node group, the time counter starts as soon as the Auto Scaling Group size increases.
## Upgrade phase
The upgrade phase behaves in two different ways, depending on the *update strategy*. There are two update strategies: **default** and **minimal**.
We recommend the default strategy in most scenarios. It creates new nodes before terminating the old ones, so that the available capacity is maintained during the upgrade phase. The minimal strategy is useful in scenarios where you are constrained to resources or costs, for example with hardware accelerators such as GPUs. It terminating the old nodes before creating the new ones, so that total capacity never increases beyond your configured quantity.
The *default* update strategy has these steps:
1. It increases the quantity of nodes (desired count) in the Auto Scaling Group, causing the node group to create additional nodes.
1. It randomly selects a node that needs to be upgraded, up to the maximum unavailable configured for the node group.
1. It drains the Pods from the node. If the Pods don’t leave the node within 15 minutes and there’s no force flag, the upgrade phase fails with a `PodEvictionFailure` error. For this scenario, you can apply the force flag with the `update-nodegroup-version` request to delete the Pods.
1. It cordons the node after every Pod is evicted and waits for 60 seconds. This is done so that the service controller doesn’t send any new requests to this node and removes this node from its list of active nodes.
1. It sends a termination request to the Auto Scaling Group for the cordoned node.
1. It repeats the previous upgrade steps until there are no nodes in the node group that are deployed with the earlier version of the launch template.
The *minimal* update strategy has these steps:
1. It cordons all nodes of the node group in the beginning, so that the service controller doesn’t send any new requests to these nodes.
1. It randomly selects a node that needs to be upgraded, up to the maximum unavailable configured for the node group.
1. It drains the Pods from the selected nodes. If the Pods don’t leave the node within 15 minutes and there’s no force flag, the upgrade phase fails with a `PodEvictionFailure` error. For this scenario, you can apply the force flag with the `update-nodegroup-version` request to delete the Pods.
1. After every Pod is evicted and waits for 60 seconds, it sends a termination request to the Auto Scaling Group for the selected nodes. The Auto Scaling Group creates new nodes (same as the number of selected nodes) to replace the missing capacity.
1. It repeats the previous upgrade steps until there are no nodes in the node group that are deployed with the earlier version of the launch template.
### `PodEvictionFailure` errors during the upgrade phase
The following are known reasons which lead to a `PodEvictionFailure` error in this phase:
**Aggressive PDB**
Aggressive PDB is defined on the Pod or there are multiple PDBs pointing to the same Pod.
**Deployment tolerating all the taints**
Once every Pod is evicted, it’s expected for the node to be empty because the node is [tainted](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) in the earlier steps. However, if the deployment tolerates every taint, then the node is more likely to be non-empty, leading to Pod eviction failure.
## Scale down phase
The scale down phase decrements the Auto Scaling group maximum size and desired size by one to return to values before the update started.
If the Upgrade workflow determines that the Cluster Autoscaler is scaling up the node group during the scale down phase of the workflow, it exits immediately without bringing the node group back to its original size.
**Note**
```
If your node group has a warm pool enabled, warm pool instances are drained before the scale-up operation begins. This is because warm pool instances have not been updated to the new launch template configuration — during the scale-up phase, they would be pulled into the Auto Scaling Group instead of launching new instances with the updated configuration, which would break the upgrade process. Draining the warm pool ensures that only new instances with the updated configuration are launched. Once the scale-down operation completes, the warm pool is restored, and the new instances in the warm pool are launched with the updated launch template configuration.
```
For more information about warm pools, see [Decrease latency for applications with long boot times using warm pools with managed node groups](warm-pools-managed-node-groups.md).
# Customize managed nodes with launch templates
For the highest level of customization, you can deploy managed nodes with your own launch template based on the steps on this page. Using a launch template allows capabilities such as to provide bootstrap arguments during deployment of a node (e.g., extra [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) arguments), assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node, deploy your own custom AMI to nodes, or deploy your own custom CNI to nodes.
When you give your own launch template upon first creating a managed node group, you will also have greater flexibility later. As long as you deploy a managed node group with your own launch template, you can iteratively update it with a different version of the same launch template. When you update your node group to a different version of your launch template, all nodes in the group are recycled to match the new configuration of the specified launch template version.
Managed node groups are always deployed with a launch template to be used with the Amazon EC2 Auto Scaling group. When you don’t provide a launch template, the Amazon EKS API creates one automatically with default values in your account. However, we don’t recommend that you modify auto-generated launch templates. Furthermore, existing node groups that don’t use a custom launch template can’t be updated directly. Instead, you must create a new node group with a custom launch template to do so.
## Launch template configuration basics
You can create an Amazon EC2 Auto Scaling launch template with the AWS Management Console, AWS CLI, or an AWS SDK. For more information, see [Creating a Launch Template for an Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/create-launch-template.html) in the *Amazon EC2 Auto Scaling User Guide*. Some of the settings in a launch template are similar to the settings used for managed node configuration. When deploying or updating a node group with a launch template, some settings must be specified in either the node group configuration or the launch template. Don’t specify a setting in both places. If a setting exists where it shouldn’t, then operations such as creating or updating a node group fail.
The following table lists the settings that are prohibited in a launch template. It also lists similar settings, if any are available, that are required in the managed node group configuration. The listed settings are the settings that appear in the console. They might have similar but different names in the AWS CLI and SDK.
| Launch template – Prohibited | Amazon EKS node group configuration |
| --- | --- |
| **Subnet** under **Network interfaces** (**Add network interface**) | **Subnets** under **Node group network configuration** on the **Specify networking** page |
| **IAM instance profile** under **Advanced details** | **Node IAM role** under **Node group configuration** on the **Configure Node group** page |
| **Shutdown behavior** and **Stop - Hibernate behavior** under **Advanced details**. Retain default **Don’t include in launch template setting** in launch template for both settings. | No equivalent. Amazon EKS must control the instance lifecycle, not the Auto Scaling group. |
The following table lists the prohibited settings in a managed node group configuration. It also lists similar settings, if any are available, which are required in a launch template. The listed settings are the settings that appear in the console. They might have similar names in the AWS CLI and SDK.
| Amazon EKS node group configuration – Prohibited | Launch template |
| --- | --- |
| (Only if you specified a custom AMI in a launch template) **AMI type** under **Node group compute configuration** on **Set compute and scaling configuration** page – Console displays **Specified in launch template** and the AMI ID that was specified. If **Application and OS Images (Amazon Machine Image)** wasn’t specified in the launch template, you can select an AMI in the node group configuration. | **Application and OS Images (Amazon Machine Image)** under **Launch template contents** – You must specify an ID if you have either of the following requirements: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html) |
| **Disk size** under **Node group compute configuration** on **Set compute and scaling configuration** page – Console displays **Specified in launch template**. | **Size** under **Storage (Volumes)** (**Add new volume**). You must specify this in the launch template. |
| **SSH key pair** under **Node group configuration** on the **Specify Networking** page – The console displays the key that was specified in the launch template or displays **Not specified in launch template**. | **Key pair name** under **Key pair (login)**. |
| You can’t specify source security groups that are allowed remote access when using a launch template. | **Security groups** under **Network settings** for the instance or **Security groups** under **Network interfaces** (**Add network interface**), but not both. For more information, see [Using custom security groups](#launch-template-security-groups). |
**Note**
If you deploy a node group using a launch template, specify zero or one **Instance type** under **Launch template contents** in a launch template. Alternatively, you can specify 0–20 instance types for **Instance types** on the **Set compute and scaling configuration** page in the console. Or, you can do so using other tools that use the Amazon EKS API. If you specify an instance type in a launch template, and use that launch template to deploy your node group, then you can’t specify any instance types in the console or using other tools that use the Amazon EKS API. If you don’t specify an instance type in a launch template, in the console, or using other tools that use the Amazon EKS API, the `t3.medium` instance type is used. If your node group is using the Spot capacity type, then we recommend specifying multiple instance types using the console. For more information, see [Managed node group capacity types](managed-node-groups.md#managed-node-group-capacity-types).
If any containers that you deploy to the node group use the Instance Metadata Service Version 2, make sure to set the **Metadata response hop limit** to `2` in your launch template. For more information, see [Instance metadata and user data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) in the *Amazon EC2 User Guide*.
Launch templates do not support the `InstanceRequirements` feature that allows flexible instance type selection.
## Tagging Amazon EC2 instances
You can use the `TagSpecification` parameter of a launch template to specify which tags to apply to Amazon EC2 instances in your node group. The IAM entity calling the `CreateNodegroup` or `UpdateNodegroupVersion` APIs must have permissions for `ec2:RunInstances` and `ec2:CreateTags`, and the tags must be added to the launch template.
## Using custom security groups
You can use a launch template to specify custom Amazon EC2 [security groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html) to apply to instances in your node group. This can be either in the instance level security groups parameter or as part of the network interface configuration parameters. However, you can’t create a launch template that specifies both instance level and network interface security groups. Consider the following conditions that apply to using custom security groups with managed node groups:
+ When using the AWS Management Console, Amazon EKS only allows launch templates with a single network interface specification.
+ By default, Amazon EKS applies the [cluster security group](sec-group-reqs.md) to the instances in your node group to facilitate communication between nodes and the control plane. If you specify custom security groups in the launch template using either option mentioned earlier, Amazon EKS doesn’t add the cluster security group. So, you must ensure that the inbound and outbound rules of your security groups enable communication with the endpoint of your cluster. If your security group rules are incorrect, the worker nodes can’t join the cluster. For more information about security group rules, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md).
+ If you need SSH access to the instances in your node group, include a security group that allows that access.
## Amazon EC2 user data
The launch template includes a section for custom user data. You can specify configuration settings for your node group in this section without manually creating individual custom AMIs. For more information about the settings available for Bottlerocket, see [Using user data](https://github.com/bottlerocket-os/bottlerocket#using-user-data) on GitHub.
You can supply Amazon EC2 user data in your launch template using `cloud-init` when launching your instances. For more information, see the [cloud-init](https://cloudinit.readthedocs.io/en/latest/index.html) documentation. Your user data can be used to perform common configuration operations. This includes the following operations:
+ [Including users or groups](https://cloudinit.readthedocs.io/en/latest/topics/examples.html#including-users-and-groups)
+ [Installing packages](https://cloudinit.readthedocs.io/en/latest/topics/examples.html#install-arbitrary-packages)
Amazon EC2 user data in launch templates that are used with managed node groups must be in the [MIME multi-part archive](https://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive) format for Amazon Linux AMIs and TOML format for Bottlerocket AMIs. This is because your user data is merged with Amazon EKS user data required for nodes to join the cluster. Don’t specify any commands in your user data that starts or modifies `kubelet`. This is performed as part of the user data merged by Amazon EKS. Certain `kubelet` parameters, such as setting labels on nodes, can be configured directly through the managed node groups API.
**Note**
For more information about advanced `kubelet` customization, including manually starting it or passing in custom configuration parameters, see [Specifying an AMI](#launch-template-custom-ami). If a custom AMI ID is specified in a launch template, Amazon EKS doesn’t merge user data.
The following details provide more information about the user data section.
**Amazon Linux 2 user data**
You can combine multiple user data blocks together into a single MIME multi-part file. For example, you can combine a cloud boothook that configures the Docker daemon with a user data shell script that installs a custom package. A MIME multi-part file consists of the following components:
+ The content type and part boundary declaration – `Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="`
+ The MIME version declaration – `MIME-Version: 1.0`
+ One or more user data blocks, which contain the following components:
+ The opening boundary, which signals the beginning of a user data block – `--==MYBOUNDARY==`
+ The content type declaration for the block: `Content-Type: text/cloud-config; charset="us-ascii"`. For more information about content types, see the [cloud-init](https://cloudinit.readthedocs.io/en/latest/topics/format.html) documentation.
+ The content of the user data (for example, a list of shell commands or `cloud-init` directives).
+ The closing boundary, which signals the end of the MIME multi-part file: `--==MYBOUNDARY==--`
The following is an example of a MIME multi-part file that you can use to create your own.
```
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
echo "Running custom user data script"
--==MYBOUNDARY==--
```
**Amazon Linux 2023 user data**
Amazon Linux 2023 (AL2023) introduces a new node initialization process `nodeadm` that uses a YAML configuration schema. If you’re using self-managed node groups or an AMI with a launch template, you’ll now need to provide additional cluster metadata explicitly when creating a new node group. An [example](https://awslabs.github.io/amazon-eks-ami/nodeadm/) of the minimum required parameters is as follows, where `apiServerEndpoint`, `certificateAuthority`, and service `cidr` are now required:
```
---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name: my-cluster
apiServerEndpoint: https://example.com
certificateAuthority: Y2VydGlmaWNhdGVBdXRob3JpdHk=
cidr: 10.100.0.0/16
```
You’ll typically set this configuration in your user data, either as-is or embedded within a MIME multi-part document:
```
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"
--BOUNDARY
Content-Type: application/node.eks.aws
---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig spec: [...]
--BOUNDARY--
```
In AL2, the metadata from these parameters was discovered from the Amazon EKS `DescribeCluster` API call. With AL2023, this behavior has changed since the additional API call risks throttling during large node scale ups. This change doesn’t affect you if you’re using managed node groups without a launch template or if you’re using Karpenter. For more information on `certificateAuthority` and service `cidr`, see [https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeCluster.html](https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeCluster.html) in the *Amazon EKS API Reference*.
Here’s a complete example of AL2023 user data that combines a shell script for customizing the node (like installing packages or pre-caching container images) with the required `nodeadm` configuration. This example shows common customizations including: \$1 Installing additional system packages \$1 Pre-caching container images to improve Pod startup time \$1 Setting up HTTP proxy configuration \$1 Configuring `kubelet` flags for node labeling
```
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"
--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
# Install additional packages
yum install -y htop jq iptables-services
# Pre-cache commonly used container images
nohup docker pull public.ecr.aws/eks-distro/kubernetes/pause:3.2 &
# Configure HTTP proxy if needed
cat > /etc/profile.d/http-proxy.sh << 'EOF'
export HTTP_PROXY="http://proxy.example.com:3128"
export HTTPS_PROXY="http://proxy.example.com:3128"
export NO_PROXY="localhost,127.0.0.1,169.254.169.254,.internal"
EOF
--BOUNDARY
Content-Type: application/node.eks.aws
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name: my-cluster
apiServerEndpoint: https://example.com
certificateAuthority: Y2VydGlmaWNhdGVBdXRob3JpdHk=
cidr: 10.100.0.0/16
kubelet:
config:
clusterDNS:
- 10.100.0.10
flags:
- --node-labels=app=my-app,environment=production
--BOUNDARY--
```
**Bottlerocket user data**
Bottlerocket structures user data in the TOML format. You can provide user data to be merged with the user data provided by Amazon EKS. For example, you can provide additional `kubelet` settings.
```
[settings.kubernetes.system-reserved]
cpu = "10m"
memory = "100Mi"
ephemeral-storage= "1Gi"
```
For more information about the supported settings, see [Bottlerocket documentation](https://github.com/bottlerocket-os/bottlerocket). You can configure node labels and [taints](node-taints-managed-node-groups.md) in your user data. However, we recommend that you configure these within your node group instead. Amazon EKS applies these configurations when you do so.
When user data is merged, formatting isn’t preserved, but the content remains the same. The configuration that you provide in your user data overrides any settings that are configured by Amazon EKS. So, if you set `settings.kubernetes.max-pods` or `settings.kubernetes.cluster-dns-ip`, these values in your user data are applied to the nodes.
Amazon EKS doesn’t support all valid TOML. The following is a list of known unsupported formats:
+ Quotes within quoted keys: `'quoted "value"' = "value"`
+ Escaped quotes in values: `str = "I’m a string. \"You can quote me\""`
+ Mixed floats and integers: `numbers = [ 0.1, 0.2, 0.5, 1, 2, 5 ]`
+ Mixed types in arrays: `contributors = ["[foo@example.com](mailto:foo@example.com)", { name = "Baz", email = "[baz@example.com](mailto:baz@example.com)" }]`
+ Bracketed headers with quoted keys: `[foo."bar.baz"]`
**Windows user data**
Windows user data uses PowerShell commands. When creating a managed node group, your custom user data combines with Amazon EKS managed user data. Your PowerShell commands come first, followed by the managed user data commands, all within one `` tag.
When creating Windows node groups, Amazon EKS updates the `aws-auth` `ConfigMap` to allow Linux-based nodes to join the cluster. The service doesn’t automatically configure permissions for Windows AMIs. If you’re using Windows nodes, you’ll need to manage access either via the access entry API or by updating the `aws-auth` `ConfigMap` directly. For more information, see [Deploy Windows nodes on EKS clusters](windows-support.md).
When no AMI ID is specified in the launch template, don’t use the Windows Amazon EKS Bootstrap script in user data to configure Amazon EKS.
Example user data is as follows.
```
Write-Host "Running custom user data script"
```
## Specifying an AMI
If you have either of the following requirements, then specify an AMI ID in the `ImageId` field of your launch template. Select the requirement you have for additional information.
### Provide user data to pass arguments to the `bootstrap.sh` file included with an Amazon EKS optimized Linux/Bottlerocket AMI
Bootstrapping is a term used to describe adding commands that can be run when an instance starts. For example, bootstrapping allows using extra [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) arguments. You can pass arguments to the `bootstrap.sh` script by using `eksctl` without specifying a launch template. Or you can do so by specifying the information in the user data section of a launch template.
**eksctl without specifying a launch template**
Create a file named *my-nodegroup.yaml* with the following contents. Replace every *example value* with your own values. The `--apiserver-endpoint`, `--b64-cluster-ca`, and `--dns-cluster-ip` arguments are optional. However, defining them allows the `bootstrap.sh` script to avoid making a `describeCluster` call. This is useful in private cluster setups or clusters where you’re scaling in and out nodes frequently. For more information on the `bootstrap.sh` script, see the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh) file on GitHub.
+ The only required argument is the cluster name (*my-cluster*).
+ To retrieve an optimized AMI ID for `ami-1234567890abcdef0 `, see the following sections:
+ [Retrieve recommended Amazon Linux AMI IDs](retrieve-ami-id.md)
+ [Retrieve recommended Bottlerocket AMI IDs](retrieve-ami-id-bottlerocket.md)
+ [Retrieve recommended Microsoft Windows AMI IDs](retrieve-windows-ami-id.md)
+ To retrieve the *certificate-authority* for your cluster, run the following command.
```
aws eks describe-cluster --query "cluster.certificateAuthority.data" --output text --name my-cluster --region region-code
```
+ To retrieve the *api-server-endpoint* for your cluster, run the following command.
```
aws eks describe-cluster --query "cluster.endpoint" --output text --name my-cluster --region region-code
```
+ The value for `--dns-cluster-ip` is your service CIDR with `.10` at the end. To retrieve the *service-cidr* for your cluster, run the following command. For example, if the returned value for is `ipv4 10.100.0.0/16`, then your value is *10.100.0.10*.
```
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
```
+ This example provides a `kubelet` argument to set a custom `max-pods` value using the `bootstrap.sh` script included with the Amazon EKS optimized AMI. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. For help with selecting *my-max-pods-value*, see [How maxPods is determined](choosing-instance-type.md#max-pods-precedence). For more information about how `maxPods` is determined when using managed node groups, see [How maxPods is determined](choosing-instance-type.md#max-pods-precedence).
```
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: my-cluster
region: region-code
managedNodeGroups:
- name: my-nodegroup
ami: ami-1234567890abcdef0
instanceType: m5.large
privateNetworking: true
disableIMDSv1: true
labels: { x86-al2-specified-mng }
overrideBootstrapCommand: |
#!/bin/bash
/etc/eks/bootstrap.sh my-cluster \
--b64-cluster-ca certificate-authority \
--apiserver-endpoint api-server-endpoint \
--dns-cluster-ip service-cidr.10 \
--kubelet-extra-args '--max-pods=my-max-pods-value' \
--use-max-pods false
```
For every available `eksctl` `config` file option, see [Config file schema](https://eksctl.io/usage/schema/) in the `eksctl` documentation. The `eksctl` utility still creates a launch template for you and populates its user data with the data that you provide in the `config` file.
Create a node group with the following command.
```
eksctl create nodegroup --config-file=my-nodegroup.yaml
```
**User data in a launch template**
Specify the following information in the user data section of your launch template. Replace every *example value* with your own values. The `--apiserver-endpoint`, `--b64-cluster-ca`, and `--dns-cluster-ip` arguments are optional. However, defining them allows the `bootstrap.sh` script to avoid making a `describeCluster` call. This is useful in private cluster setups or clusters where you’re scaling in and out nodes frequently. For more information on the `bootstrap.sh` script, see the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh) file on GitHub.
+ The only required argument is the cluster name (*my-cluster*).
+ To retrieve the *certificate-authority* for your cluster, run the following command.
```
aws eks describe-cluster --query "cluster.certificateAuthority.data" --output text --name my-cluster --region region-code
```
+ To retrieve the *api-server-endpoint* for your cluster, run the following command.
```
aws eks describe-cluster --query "cluster.endpoint" --output text --name my-cluster --region region-code
```
+ The value for `--dns-cluster-ip` is your service CIDR with `.10` at the end. To retrieve the *service-cidr* for your cluster, run the following command. For example, if the returned value for is `ipv4 10.100.0.0/16`, then your value is *10.100.0.10*.
```
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
```
+ This example provides a `kubelet` argument to set a custom `max-pods` value using the `bootstrap.sh` script included with the Amazon EKS optimized AMI. For help with selecting *my-max-pods-value*, see [How maxPods is determined](choosing-instance-type.md#max-pods-precedence). For more information about how `maxPods` is determined when using managed node groups, see [How maxPods is determined](choosing-instance-type.md#max-pods-precedence).
```
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
/etc/eks/bootstrap.sh my-cluster \
--b64-cluster-ca certificate-authority \
--apiserver-endpoint api-server-endpoint \
--dns-cluster-ip service-cidr.10 \
--kubelet-extra-args '--max-pods=my-max-pods-value' \
--use-max-pods false
--==MYBOUNDARY==--
```
### Provide user data to pass arguments to the `Start-EKSBootstrap.ps1` file included with an Amazon EKS optimized Windows AMI
Bootstrapping is a term used to describe adding commands that can be run when an instance starts. You can pass arguments to the `Start-EKSBootstrap.ps1` script by using `eksctl` without specifying a launch template. Or you can do so by specifying the information in the user data section of a launch template.
If you want to specify a custom Windows AMI ID, keep in mind the following considerations:
+ You must use a launch template and give the required bootstrap commands in the user data section. To retrieve your desired Windows ID, you can use the table in [Create nodes with optimized Windows AMIs](eks-optimized-windows-ami.md).
+ There are several limits and conditions. For example, you must add `eks:kube-proxy-windows` to your AWS IAM Authenticator configuration map. For more information, see [Limits and conditions when specifying an AMI ID](#mng-ami-id-conditions).
Specify the following information in the user data section of your launch template. Replace every *example value* with your own values. The `-APIServerEndpoint`, `-Base64ClusterCA`, and `-DNSClusterIP` arguments are optional. However, defining them allows the `Start-EKSBootstrap.ps1` script to avoid making a `describeCluster` call.
+ The only required argument is the cluster name (*my-cluster*).
+ To retrieve the *certificate-authority* for your cluster, run the following command.
```
aws eks describe-cluster --query "cluster.certificateAuthority.data" --output text --name my-cluster --region region-code
```
+ To retrieve the *api-server-endpoint* for your cluster, run the following command.
```
aws eks describe-cluster --query "cluster.endpoint" --output text --name my-cluster --region region-code
```
+ The value for `--dns-cluster-ip` is your service CIDR with `.10` at the end. To retrieve the *service-cidr* for your cluster, run the following command. For example, if the returned value for is `ipv4 10.100.0.0/16`, then your value is *10.100.0.10*.
```
aws eks describe-cluster --query "cluster.kubernetesNetworkConfig.serviceIpv4Cidr" --output text --name my-cluster --region region-code
```
+ For additional arguments, see [Bootstrap script configuration parameters](eks-optimized-windows-ami.md#bootstrap-script-configuration-parameters).
**Note**
If you’re using custom service CIDR, then you need to specify it using the `-ServiceCIDR` parameter. Otherwise, the DNS resolution for Pods in the cluster will fail.
```
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName my-cluster `
-Base64ClusterCA certificate-authority `
-APIServerEndpoint api-server-endpoint `
-DNSClusterIP service-cidr.10
```
### Run a custom AMI due to specific security, compliance, or internal policy requirements
For more information, see [Amazon Machine Images (AMI)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) in the *Amazon EC2 User Guide*. The Amazon EKS AMI build specification contains resources and configuration scripts for building a custom Amazon EKS AMI based on Amazon Linux. For more information, see [Amazon EKS AMI Build Specification](https://github.com/awslabs/amazon-eks-ami/) on GitHub. To build custom AMIs installed with other operating systems, see [Amazon EKS Sample Custom AMIs](https://github.com/aws-samples/amazon-eks-custom-amis) on GitHub.
You cannot use dynamic parameter references for AMI IDs in Launch Templates used with managed node groups.
**Important**
When specifying an AMI, Amazon EKS does not validate the Kubernetes version embedded in your AMI against your cluster’s control plane version. You are responsible for ensuring that the Kubernetes version of your custom AMI conforms to the [Kubernetes version skew policy](https://kubernetes.io/releases/version-skew-policy):
The `kubelet` version on your nodes must not be newer than your cluster version
The `kubelet` version on your nodes must be equal to or up to 3 minor versions behind your cluster version (for Kubernetes version `1.28` or higher), or up to 2 minor versions behind your cluster version (for Kubernetes version `1.27` or lower)
Creating managed node groups with version skew violations may result in:
Nodes failing to join the cluster
Undefined behavior or API incompatibilities
Cluster instability or workload failures
When specifying an AMI, Amazon EKS doesn’t merge any user data. Rather, you’re responsible for supplying the required `bootstrap` commands for nodes to join the cluster. If your nodes fail to join the cluster, the Amazon EKS `CreateNodegroup` and `UpdateNodegroupVersion` actions also fail.
## Limits and conditions when specifying an AMI ID
The following are the limits and conditions involved with specifying an AMI ID with managed node groups:
+ You must create a new node group to switch between specifying an AMI ID in a launch template and not specifying an AMI ID.
+ You aren’t notified in the console when a newer AMI version is available. To update your node group to a newer AMI version, you need to create a new version of your launch template with an updated AMI ID. Then, you need to update the node group with the new launch template version.
+ The following fields can’t be set in the API if you specify an AMI ID:
+ `amiType`
+ `releaseVersion`
+ `version`
+ Any `taints` set in the API are applied asynchronously if you specify an AMI ID. To apply taints prior to a node joining the cluster, you must pass the taints to `kubelet` in your user data using the `--register-with-taints` command line flag. For more information, see [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) in the Kubernetes documentation.
+ When specifying a custom AMI ID for Windows managed node groups, add `eks:kube-proxy-windows` to your AWS IAM Authenticator configuration map. This is required for DNS to function properly.
1. Open the AWS IAM Authenticator configuration map for editing.
```
kubectl edit -n kube-system cm aws-auth
```
1. Add this entry to the `groups` list under each `rolearn` associated with Windows nodes. Your configuration map should look similar to [aws-auth-cm-windows.yaml](https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/aws-auth-cm-windows.yaml).
```
- eks:kube-proxy-windows
```
1. Save the file and exit your text editor.
+ For any AMI that uses a custom launch template, the default `HttpPutResponseHopLimit` for managed node groups is set to `2`.
# Delete a managed node group from your cluster
This topic describes how you can delete an Amazon EKS managed node group. When you delete a managed node group, Amazon EKS first sets the minimum, maximum, and desired size of your Auto Scaling group to zero. This then causes your node group to scale down.
Before each instance is terminated, Amazon EKS sends a signal to drain that node. During the drain process, Kubernetes does the following for each pod on the node: runs any configured `preStop` lifecycle hooks, sends `SIGTERM` signals to the containers, then waits for the `terminationGracePeriodSeconds` for graceful shutdown. If the node hasn’t been drained after 5 minutes, Amazon EKS lets Auto Scaling continue the forced termination of the instance. After all instances have been terminated, the Auto Scaling group is deleted.
**Important**
If you delete a managed node group that uses a node IAM role that isn’t used by any other managed node group in the cluster, the role is removed from the `aws-auth` `ConfigMap`. If any of the self-managed node groups in the cluster are using the same node IAM role, the self-managed nodes move to the `NotReady` status. Additionally, the cluster operation is also disrupted. To add a mapping for the role you’re using only for the self-managed node groups, see [Create access entries](creating-access-entries.md), if your cluster’s platform version is at least minimum version listed in the prerequisites section of [Grant IAM users access to Kubernetes with EKS access entries](access-entries.md). If your platform version is earlier than the required minimum version for access entries, you can add the entry back to the `aws-auth` `ConfigMap`. For more information, enter `eksctl create iamidentitymapping --help` in your terminal.
You can delete a managed node group with:
+ [`eksctl`](#eksctl-delete-managed-nodegroup)
+ [AWS Management Console](#console-delete-managed-nodegroup)
+ [AWS CLI](#awscli-delete-managed-nodegroup)
## `eksctl`
**Delete a managed node group with `eksctl` **
Enter the following command. Replace every `` with your own values.
```
eksctl delete nodegroup \
--cluster \
--name \
--region
```
For more options, see [Deleting and draining nodegroups](https://eksctl.io/usage/nodegroups/#deleting-and-draining-nodegroups) in the `eksctl` documentation.
## AWS Management Console
**Delete a managed node group with AWS Management Console **
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. On the **Clusters** page, choose the cluster that contains the node group to delete.
1. On the selected cluster page, choose the **Compute** tab.
1. In the **Node groups** section, choose the node group to delete. Then choose **Delete**.
1. In the **Delete node group** confirmation dialog box, enter the name of the node group. Then choose **Delete**.
## AWS CLI
**Delete a managed node group with AWS CLI**
1. Enter the following command. Replace every `` with your own values.
```
aws eks delete-nodegroup \
--cluster-name \
--nodegroup-name \
--region
```
1. If `cli_pager=` is set in the CLI config, use the arrow keys on your keyboard to scroll through the response output. Press the `q` key when you’re finished.
For more options, see the ` [delete-nodegroup](https://docs.aws.amazon.com/cli/latest/reference/eks/delete-nodegroup.html) ` command in the * AWS CLI Command Reference*.
# Maintain nodes yourself with self-managed nodes
A cluster contains one or more Amazon EC2 nodes that Pods are scheduled on. Amazon EKS nodes run in your AWS account and connect to the control plane of your cluster through the cluster API server endpoint. You’re billed for them based on Amazon EC2 prices. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/).
A cluster can contain several node groups. Each node group contains one or more nodes that are deployed in an [Amazon EC2 Auto Scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html). The instance type of the nodes within the group can vary, such as when using [attribute-based instance type selection](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-fleet-attribute-based-instance-type-selection.html) with [Karpenter](https://karpenter.sh/). All instances in a node group must use the [Amazon EKS node IAM role](create-node-role.md).
Amazon EKS provides specialized Amazon Machine Images (AMIs) that are called Amazon EKS optimized AMIs. The AMIs are configured to work with Amazon EKS. Their components include `containerd`, `kubelet`, and the AWS IAM Authenticator. The AMIs also contain a specialized [bootstrap script](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh) that allows it to discover and connect to your cluster’s control plane automatically.
If you restrict access to the public endpoint of your cluster using CIDR blocks, we recommend that you also enable private endpoint access. This is so that nodes can communicate with the cluster. Without the private endpoint enabled, the CIDR blocks that you specify for public access must include the egress sources from your VPC. For more information, see [Cluster API server endpoint](cluster-endpoint.md).
To add self-managed nodes to your Amazon EKS cluster, see the topics that follow. If you launch self-managed nodes manually, add the following tag to each node while making sure that `` matches your cluster. For more information, see [Adding and deleting tags on an individual resource](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#adding-or-deleting-tags). If you follow the steps in the guides that follow, the required tag is automatically added to nodes for you.
| Key | Value |
| --- | --- |
| `kubernetes.io/cluster/` | `owned` |
**Important**
Tags in Amazon EC2 Instance Metadata Service (IMDS) are not compatible with EKS nodes. When Instance Metadata Tags are enabled, the use of forward slashes ('/') in tag values is prevented. This limitation can cause instance launch failures, particularly when using node management tools like Karpenter or Cluster Autoscaler, as these services rely on tags containing forward slashes for proper functionality.
For more information about nodes from a general Kubernetes perspective, see [Nodes](https://kubernetes.io/docs/concepts/architecture/nodes/) in the Kubernetes documentation.
**Topics**
+ [
# Create self-managed Amazon Linux nodes
](launch-workers.md)
+ [
# Create self-managed Bottlerocket nodes
](launch-node-bottlerocket.md)
+ [
# Create self-managed Microsoft Windows nodes
](launch-windows-workers.md)
+ [
# Create self-managed Ubuntu Linux nodes
](launch-node-ubuntu.md)
+ [
# Update self-managed nodes for your cluster
](update-workers.md)
# Create self-managed Amazon Linux nodes
This topic describes how you can launch Auto Scaling groups of Linux nodes that register with your Amazon EKS cluster. After the nodes join the cluster, you can deploy Kubernetes applications to them. You can also launch self-managed Amazon Linux nodes with `eksctl` or the AWS Management Console. If you need to launch nodes on AWS Outposts, see [Create Amazon Linux nodes on AWS Outposts](eks-outposts-self-managed-nodes.md).
+ An existing Amazon EKS cluster. To deploy one, see [Create an Amazon EKS cluster](create-cluster.md). If you have subnets in the AWS Region where you have AWS Outposts, AWS Wavelength, or AWS Local Zones enabled, those subnets must not have been passed in when you created your cluster.
+ An existing IAM role for the nodes to use. To create one, see [Amazon EKS node IAM role](create-node-role.md). If this role doesn’t have either of the policies for the VPC CNI, the separate role that follows is required for the VPC CNI pods.
+ (Optional, but recommended) The Amazon VPC CNI plugin for Kubernetes add-on configured with its own IAM role that has the necessary IAM policy attached to it. For more information, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md).
+ Familiarity with the considerations listed in [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md). Depending on the instance type you choose, there may be additional prerequisites for your cluster and VPC.
You can launch self-managed Linux nodes using either of the following:
+ [`eksctl`](#eksctl_create_managed_amazon_linux)
+ [AWS Management Console](#console_create_managed_amazon_linux)
## `eksctl`
**Launch self-managed Linux nodes using `eksctl` **
1. Install version `0.215.0` or later of the `eksctl` command line tool installed on your device or AWS CloudShell. To install or update `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
1. (Optional) If the **AmazonEKS\$1CNI\$1Policy** managed IAM policy is attached to your [Amazon EKS node IAM role](create-node-role.md), we recommend assigning it to an IAM role that you associate to the Kubernetes `aws-node` service account instead. For more information, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md).
1. The following command creates a node group in an existing cluster. Replace *al-nodes* with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. Replace *my-cluster* with the name of your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in. Replace the remaining *example value* with your own values. The nodes are created with the same Kubernetes version as the control plane, by default.
Before choosing a value for `--node-type`, review [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md).
Replace *my-key* with the name of your Amazon EC2 key pair or public key. This key is used to SSH into your nodes after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*.
Create your node group with the following command.
**Important**
If you want to deploy a node group to AWS Outposts, Wavelength, or Local Zone subnets, there are additional considerations:
The subnets must not have been passed in when you created the cluster.
You must create the node group with a config file that specifies the subnets and ` [volumeType](https://eksctl.io/usage/schema/#nodeGroups-volumeType): gp2`. For more information, see [Create a nodegroup from a config file](https://eksctl.io/usage/nodegroups/#creating-a-nodegroup-from-a-config-file) and [Config file schema](https://eksctl.io/usage/schema/) in the `eksctl` documentation.
```
eksctl create nodegroup \
--cluster my-cluster \
--name al-nodes \
--node-type t3.medium \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--ssh-access \
--managed=false \
--ssh-public-key my-key
```
To deploy a node group that:
+ can assign a significantly higher number of IP addresses to Pods than the default configuration, see [Assign more IP addresses to Amazon EKS nodes with prefixes](cni-increase-ip-addresses.md).
+ can assign `IPv4` addresses to Pods from a different CIDR block than that of the instance, see [Deploy Pods in alternate subnets with custom networking](cni-custom-network.md).
+ can assign `IPv6` addresses to Pods and services, see [Learn about IPv6 addresses to clusters, Pods, and services](cni-ipv6.md).
+ don’t have outbound internet access, see [Deploy private clusters with limited internet access](private-clusters.md).
For a complete list of all available options and defaults, enter the following command.
```
eksctl create nodegroup --help
```
If nodes fail to join the cluster, then see [Nodes fail to join cluster](troubleshooting.md#worker-node-fail) in the Troubleshooting chapter.
An example output is as follows. Several lines are output while the nodes are created. One of the last lines of output is the following example line.
```
[✔] created 1 nodegroup(s) in cluster "my-cluster"
```
1. (Optional) Deploy a [sample application](sample-deployment.md) to test your cluster and Linux nodes.
1. We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
## AWS Management Console
**Step 1: Launch self-managed Linux nodes using AWS Management Console **
1. Download the latest version of the AWS CloudFormation template.
```
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2025-11-26/amazon-eks-nodegroup.yaml
```
1. Wait for your cluster status to show as `ACTIVE`. If you launch your nodes before the cluster is active, the nodes fail to register with the cluster and you will have to relaunch them.
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Choose **Create stack** and then select **With new resources (standard)**.
1. For **Specify template**, select **Upload a template file** and then select **Choose file**.
1. Select the `amazon-eks-nodegroup.yaml` file that you downloaded.
1. Select **Next**.
1. On the **Specify stack details** page, enter the following parameters accordingly, and then choose **Next**:
+ **Stack name**: Choose a stack name for your AWS CloudFormation stack. For example, you can call it *my-cluster-nodes*. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
+ **ClusterName**: Enter the name that you used when you created your Amazon EKS cluster. This name must equal the cluster name or your nodes can’t join the cluster.
+ **ClusterControlPlaneSecurityGroup**: Choose the **SecurityGroups** value from the AWS CloudFormation output that you generated when you created your [VPC](creating-a-vpc.md).
The following steps show one operation to retrieve the applicable group.
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the name of the cluster.
1. Choose the **Networking** tab.
1. Use the **Additional security groups** value as a reference when selecting from the **ClusterControlPlaneSecurityGroup** dropdown list.
+ **ApiServerEndpoint**: Enter the API Server Endpoint for your EKS Cluster. This can be found in the Details section of the EKS Cluster Console
+ **CertificateAuthorityData**: Enter the base64 encoded Certificate Authority data which can also be found in the EKS Cluster Console’s Details section.
+ **ServiceCidr**: Enter the CIDR range used for allocating IP addresses to Kubernetes services within the cluster. This is found within the networking tab of the EKS Cluster Console.
+ **AuthenticationMode**: Select the Authentication Mode in use in the EKS Cluster by reviewing the access tab within the EKS Cluster Console.
+ **NodeGroupName**: Enter a name for your node group. This name can be used later to identify the Auto Scaling node group that’s created for your nodes. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
+ **NodeAutoScalingGroupMinSize**: Enter the minimum number of nodes that your node Auto Scaling group can scale in to.
+ **NodeAutoScalingGroupDesiredCapacity**: Enter the desired number of nodes to scale to when your stack is created.
+ **NodeAutoScalingGroupMaxSize**: Enter the maximum number of nodes that your node Auto Scaling group can scale out to.
+ **NodeInstanceType**: Choose an instance type for your nodes. For more information, see [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md).
+ **NodeImageIdSSMParam**: Pre-populated with the Amazon EC2 Systems Manager parameter of a recent Amazon EKS optimized Amazon Linux 2023 AMI for a variable Kubernetes version. To use a different Kubernetes minor version supported with Amazon EKS, replace *1.XX* with a different [supported version](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). We recommend specifying the same Kubernetes version as your cluster.
You can also replace *amazon-linux-2023* with a different AMI type. For more information, see [Retrieve recommended Amazon Linux AMI IDs](retrieve-ami-id.md).
**Note**
The Amazon EKS node AMIs are based on Amazon Linux. You can track security or privacy events for Amazon Linux 2023 at the [Amazon Linux Security Center](https://alas.aws.amazon.com/alas2023.html) or subscribe to the associated [RSS feed](https://alas.aws.amazon.com/AL2023/alas.rss). Security and privacy events include an overview of the issue, what packages are affected, and how to update your instances to correct the issue.
+ **NodeImageId**: (Optional) If you’re using your own custom AMI (instead of an Amazon EKS optimized AMI), enter a node AMI ID for your AWS Region. If you specify a value here, it overrides any values in the **NodeImageIdSSMParam** field.
+ **NodeVolumeSize**: Specify a root volume size for your nodes, in GiB.
+ **NodeVolumeType**: Specify a root volume type for your nodes.
+ **KeyName**: Enter the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*.
+ **VpcId**: Enter the ID for the [VPC](creating-a-vpc.md) that you created.
+ **Subnets**: Choose the subnets that you created for your VPC. If you created your VPC using the steps that are described in [Create an Amazon VPC for your Amazon EKS cluster](creating-a-vpc.md), specify only the private subnets within the VPC for your nodes to launch into. You can see which subnets are private by opening each subnet link from the **Networking** tab of your cluster.
**Important**
If any of the subnets are public subnets, then they must have the automatic public IP address assignment setting enabled. If the setting isn’t enabled for the public subnet, then any nodes that you deploy to that public subnet won’t be assigned a public IP address and won’t be able to communicate with the cluster or other AWS services. If the subnet was deployed before March 26, 2020 using either of the [Amazon EKS AWS CloudFormation VPC templates](creating-a-vpc.md), or by using `eksctl`, then automatic public IP address assignment is disabled for public subnets. For information about how to enable public IP address assignment for a subnet, see [Modifying the public IPv4 addressing attribute for your subnet](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#subnet-public-ip). If the node is deployed to a private subnet, then it’s able to communicate with the cluster and other AWS services through a NAT gateway.
If the subnets don’t have internet access, make sure that you’re aware of the considerations and extra steps in [Deploy private clusters with limited internet access](private-clusters.md).
If you select AWS Outposts, Wavelength, or Local Zone subnets, the subnets must not have been passed in when you created the cluster.
1. Select your desired choices on the **Configure stack options** page, and then choose **Next**.
1. Select the check box to the left of **I acknowledge that AWS CloudFormation might create IAM resources.**, and then choose **Create stack**.
1. When your stack has finished creating, select it in the console and choose **Outputs**. If you are using the `EKS API` or `EKS API and ConfigMap` Authentication Modes, this is the last step.
1. If you are using the `ConfigMap` Authentication Mode, record the **NodeInstanceRole** for the node group that was created.
**Step 2: Enable nodes to join your cluster**
**Note**
The following two steps are only needed if using the Configmap Authentication Mode within the EKS Cluster. Additionally, if you launched nodes inside a private VPC without outbound internet access, make sure to enable nodes to join your cluster from within the VPC.
1. Check to see if you already have an `aws-auth` `ConfigMap`.
```
kubectl describe configmap -n kube-system aws-auth
```
1. If you are shown an `aws-auth` `ConfigMap`, then update it as needed.
1. Open the `ConfigMap` for editing.
```
kubectl edit -n kube-system configmap/aws-auth
```
1. Add a new `mapRoles` entry as needed. Set the `rolearn` value to the **NodeInstanceRole** value that you recorded in the previous procedure.
```
[...]
data:
mapRoles: |
- rolearn:
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
[...]
```
1. Save the file and exit your text editor.
1. If you received an error stating "`Error from server (NotFound): configmaps "aws-auth" not found`, then apply the stack `ConfigMap`.
1. Download the configuration map.
```
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/aws-auth-cm.yaml
```
1. In the `aws-auth-cm.yaml` file, set the `rolearn` value to the **NodeInstanceRole** value that you recorded in the previous procedure. You can do this with a text editor, or by replacing *my-node-instance-role* and running the following command:
```
sed -i.bak -e 's||my-node-instance-role|' aws-auth-cm.yaml
```
1. Apply the configuration. This command may take a few minutes to finish.
```
kubectl apply -f aws-auth-cm.yaml
```
1. Watch the status of your nodes and wait for them to reach the `Ready` status.
```
kubectl get nodes --watch
```
Enter `Ctrl`\$1`C` to return to a shell prompt.
**Note**
If you receive any authorization or resource type errors, see [Unauthorized or access denied (`kubectl`)](troubleshooting.md#unauthorized) in the troubleshooting topic.
If nodes fail to join the cluster, then see [Nodes fail to join cluster](troubleshooting.md#worker-node-fail) in the Troubleshooting chapter.
1. (GPU nodes only) If you chose a GPU instance type and the Amazon EKS optimized accelerated AMI, you must apply the [NVIDIA device plugin for Kubernetes](https://github.com/NVIDIA/k8s-device-plugin) as a DaemonSet on your cluster. Replace *vX.X.X* with your desired [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin/releases) version before running the following command.
```
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/vX.X.X/deployments/static/nvidia-device-plugin.yml
```
**Step 3: Additional actions**
1. (Optional) Deploy a [sample application](sample-deployment.md) to test your cluster and Linux nodes.
1. (Optional) If the **AmazonEKS\$1CNI\$1Policy** managed IAM policy (if you have an `IPv4` cluster) or the *AmazonEKS\$1CNI\$1IPv6\$1Policy* (that you [created yourself](cni-iam-role.md#cni-iam-role-create-ipv6-policy) if you have an `IPv6` cluster) is attached to your [Amazon EKS node IAM role](create-node-role.md), we recommend assigning it to an IAM role that you associate to the Kubernetes `aws-node` service account instead. For more information, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md).
1. We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
# Create self-managed Bottlerocket nodes
**Note**
Managed node groups might offer some advantages for your use case. For more information, see [Simplify node lifecycle with managed node groups](managed-node-groups.md).
This topic describes how to launch Auto Scaling groups of [Bottlerocket](https://aws.amazon.com/bottlerocket/) nodes that register with your Amazon EKS cluster. Bottlerocket is a Linux-based open-source operating system from AWS that you can use for running containers on virtual machines or bare metal hosts. After the nodes join the cluster, you can deploy Kubernetes applications to them. For more information about Bottlerocket, see [Using a Bottlerocket AMI with Amazon EKS](https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART-EKS.md) on GitHub and [Custom AMI support](https://eksctl.io/usage/custom-ami-support/) in the `eksctl` documentation.
For information about in-place upgrades, see [Bottlerocket Update Operator](https://github.com/bottlerocket-os/bottlerocket-update-operator) on GitHub.
**Important**
Amazon EKS nodes are standard Amazon EC2 instances, and you are billed for them based on normal Amazon EC2 instance prices. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/).
You can launch Bottlerocket nodes in Amazon EKS extended clusters on AWS Outposts, but you can’t launch them in local clusters on AWS Outposts. For more information, see [Deploy Amazon EKS on-premises with AWS Outposts](eks-outposts.md).
You can deploy to Amazon EC2 instances with `x86` or Arm processors. However, you can’t deploy to instances that have Inferentia chips.
Bottlerocket is compatible with AWS CloudFormation. However, there is no official CloudFormation template that can be copied to deploy Bottlerocket nodes for Amazon EKS.
Bottlerocket images don’t come with an SSH server or a shell. You can use out-of-band access methods to allow SSH enabling the admin container and to pass some bootstrapping configuration steps with user data. For more information, see these sections in the [bottlerocket README.md](https://github.com/bottlerocket-os/bottlerocket) on GitHub:
[Exploration](https://github.com/bottlerocket-os/bottlerocket#exploration)
[Admin container](https://github.com/bottlerocket-os/bottlerocket#admin-container)
[Kubernetes settings](https://github.com/bottlerocket-os/bottlerocket#kubernetes-settings)
This procedure requires `eksctl` version `0.215.0` or later. You can check your version with the following command:
```
eksctl version
```
For instructions on how to install or upgrade `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.NOTE: This procedure only works for clusters that were created with `eksctl`.
1. Copy the following contents to your device. Replace *my-cluster* with the name of your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in. Replace *ng-bottlerocket* with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. To deploy on Arm instances, replace *m5.large* with an Arm instance type. Replace *my-ec2-keypair-name* with the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*. Replace all remaining example values with your own values. Once you’ve made the replacements, run the modified command to create the `bottlerocket.yaml` file.
If specifying an Arm Amazon EC2 instance type, then review the considerations in [Amazon EKS optimized Arm Amazon Linux AMIs](eks-optimized-ami.md#arm-ami) before deploying. For instructions on how to deploy using a custom AMI, see [Building Bottlerocket](https://github.com/bottlerocket-os/bottlerocket/blob/develop/BUILDING.md) on GitHub and [Custom AMI support](https://eksctl.io/usage/custom-ami-support/) in the `eksctl` documentation. To deploy a managed node group, deploy a custom AMI using a launch template. For more information, see [Customize managed nodes with launch templates](launch-templates.md).
**Important**
To deploy a node group to AWS Outposts, AWS Wavelength, or AWS Local Zone subnets, don’t pass AWS Outposts, AWS Wavelength, or AWS Local Zone subnets when you create the cluster. You must specify the subnets in the following example. For more information see [Create a nodegroup from a config file](https://eksctl.io/usage/nodegroups/#creating-a-nodegroup-from-a-config-file) and [Config file schema](https://eksctl.io/usage/schema/) in the `eksctl` documentation. Replace *region-code* with the AWS Region that your cluster is in.
```
cat >bottlerocket.yaml <
This topic describes how to launch Auto Scaling groups of Windows nodes that register with your Amazon EKS cluster. After the nodes join the cluster, you can deploy Kubernetes applications to them.
**Important**
Amazon EKS nodes are standard Amazon EC2 instances, and you are billed for them based on normal Amazon EC2 instance prices. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/).
You can launch Windows nodes in Amazon EKS extended clusters on AWS Outposts, but you can’t launch them in local clusters on AWS Outposts. For more information, see [Deploy Amazon EKS on-premises with AWS Outposts](eks-outposts.md).
Enable Windows support for your cluster. We recommend that you review important considerations before you launch a Windows node group. For more information, see [Enable Windows support](windows-support.md#enable-windows-support).
You can launch self-managed Windows nodes with either of the following:
+ [`eksctl`](#eksctl_create_windows_nodes)
+ [AWS Management Console](#console_create_windows_nodes)
## `eksctl`
**Launch self-managed Windows nodes using `eksctl` **
This procedure requires that you have installed `eksctl`, and that your `eksctl` version is at least `0.215.0`. You can check your version with the following command.
```
eksctl version
```
For instructions on how to install or upgrade `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
**Note**
This procedure only works for clusters that were created with `eksctl`.
1. (Optional) If the **AmazonEKS\$1CNI\$1Policy** managed IAM policy (if you have an `IPv4` cluster) or the *AmazonEKS\$1CNI\$1IPv6\$1Policy* (that you [created yourself](cni-iam-role.md#cni-iam-role-create-ipv6-policy) if you have an `IPv6` cluster) is attached to your [Amazon EKS node IAM role](create-node-role.md), we recommend assigning it to an IAM role that you associate to the Kubernetes `aws-node` service account instead. For more information, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md).
1. This procedure assumes that you have an existing cluster. If you don’t already have an Amazon EKS cluster and an Amazon Linux node group to add a Windows node group to, we recommend that you follow [Get started with Amazon EKS – `eksctl`](getting-started-eksctl.md). This guide provides a complete walkthrough for how to create an Amazon EKS cluster with Amazon Linux nodes.
Create your node group with the following command. Replace *region-code* with the AWS Region that your cluster is in. Replace *my-cluster* with your cluster name. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in. Replace *ng-windows* with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. You can replace *2019* with `2022` to use Windows Server 2022 or `2025` to use Windows Server 2025. Replace the rest of the example values with your own values.
**Important**
To deploy a node group to AWS Outposts, AWS Wavelength, or AWS Local Zone subnets, don’t pass the AWS Outposts, Wavelength, or Local Zone subnets when you create the cluster. Create the node group with a config file, specifying the AWS Outposts, Wavelength, or Local Zone subnets. For more information, see [Create a nodegroup from a config file](https://eksctl.io/usage/nodegroups/#creating-a-nodegroup-from-a-config-file) and [Config file schema](https://eksctl.io/usage/schema/) in the `eksctl` documentation.
```
eksctl create nodegroup \
--region region-code \
--cluster my-cluster \
--name ng-windows \
--node-type t2.large \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--managed=false \
--node-ami-family WindowsServer2019FullContainer
```
**Note**
If nodes fail to join the cluster, see [Nodes fail to join cluster](troubleshooting.md#worker-node-fail) in the Troubleshooting guide.
To see the available options for `eksctl` commands, enter the following command.
```
eksctl command -help
```
An example output is as follows. Several lines are output while the nodes are created. One of the last lines of output is the following example line.
```
[✔] created 1 nodegroup(s) in cluster "my-cluster"
```
1. (Optional) Deploy a [sample application](sample-deployment.md) to test your cluster and Windows nodes.
1. We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
## AWS Management Console
**Prerequisites**
+ An existing Amazon EKS cluster and a Linux node group. If you don’t have these resources, we recommend that you create them using one of our guides in [Get started with Amazon EKS](getting-started.md). These guides describe how to create an Amazon EKS cluster with Linux nodes.
+ An existing VPC and security group that meet the requirements for an Amazon EKS cluster. For more information, see [View Amazon EKS networking requirements for VPC and subnets](network-reqs.md) and [View Amazon EKS security group requirements for clusters](sec-group-reqs.md). The guides in [Get started with Amazon EKS](getting-started.md) create a VPC that meets the requirements. Alternatively, you can also follow [Create an Amazon VPC for your Amazon EKS cluster](creating-a-vpc.md) to create one manually.
+ An existing Amazon EKS cluster that uses a VPC and security group that meets the requirements of an Amazon EKS cluster. For more information, see [Create an Amazon EKS cluster](create-cluster.md). If you have subnets in the AWS Region where you have AWS Outposts, AWS Wavelength, or AWS Local Zones enabled, those subnets must not have been passed in when you created the cluster.
**Step 1: Launch self-managed Windows nodes using the AWS Management Console **
1. Wait for your cluster status to show as `ACTIVE`. If you launch your nodes before the cluster is active, the nodes fail to register with the cluster and you need to relaunch them.
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/)
1. Choose **Create stack**.
1. For **Specify template**, select **Amazon S3 URL**.
1. Copy the following URL and paste it into **Amazon S3 URL**.
```
https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2023-02-09/amazon-eks-windows-nodegroup.yaml
```
1. Select **Next** twice.
1. On the **Quick create stack** page, enter the following parameters accordingly:
+ **Stack name**: Choose a stack name for your AWS CloudFormation stack. For example, you can call it `my-cluster-nodes`.
+ **ClusterName**: Enter the name that you used when you created your Amazon EKS cluster.
**Important**
This name must exactly match the name that you used in [Step 1: Create your Amazon EKS cluster](getting-started-console.md#eks-create-cluster). Otherwise, your nodes can’t join the cluster.
+ **ClusterControlPlaneSecurityGroup**: Choose the security group from the AWS CloudFormation output that you generated when you created your [VPC](creating-a-vpc.md). The following steps show one method to retrieve the applicable group.
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the name of the cluster.
1. Choose the **Networking** tab.
1. Use the **Additional security groups** value as a reference when selecting from the **ClusterControlPlaneSecurityGroup** dropdown list.
+ **NodeGroupName**: Enter a name for your node group. This name can be used later to identify the Auto Scaling node group that’s created for your nodes. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters.
+ **NodeAutoScalingGroupMinSize**: Enter the minimum number of nodes that your node Auto Scaling group can scale in to.
+ **NodeAutoScalingGroupDesiredCapacity**: Enter the desired number of nodes to scale to when your stack is created.
+ **NodeAutoScalingGroupMaxSize**: Enter the maximum number of nodes that your node Auto Scaling group can scale out to.
+ **NodeInstanceType**: Choose an instance type for your nodes. For more information, see [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md).
**Note**
The supported instance types for the latest version of the [Amazon VPC CNI plugin for Kubernetes](https://github.com/aws/amazon-vpc-cni-k8s) are listed in [vpc\$1ip\$1resource\$1limit.go](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/pkg/vpc/vpc_ip_resource_limit.go) on GitHub. You might need to update your CNI version to use the latest supported instance types. For more information, see [Assign IPs to Pods with the Amazon VPC CNI](managing-vpc-cni.md).
+ **NodeImageIdSSMParam**: Pre-populated with the Amazon EC2 Systems Manager parameter of the current recommended Amazon EKS optimized Windows Core AMI ID. To use the full version of Windows, replace *Core* with `Full`.
+ **NodeImageId**: (Optional) If you’re using your own custom AMI (instead of an Amazon EKS optimized AMI), enter a node AMI ID for your AWS Region. If you specify a value for this field, it overrides any values in the **NodeImageIdSSMParam** field.
+ **NodeVolumeSize**: Specify a root volume size for your nodes, in GiB.
+ **KeyName**: Enter the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-key-pairs.html) in the *Amazon EC2 User Guide*.
**Note**
If you don’t provide a key pair here, the AWS CloudFormation stack fails to be created.
+ **BootstrapArguments**: Specify any optional arguments to pass to the node bootstrap script, such as extra `kubelet` arguments using `-KubeletExtraArgs`.
+ **DisableIMDSv1**: By default, each node supports the Instance Metadata Service Version 1 (IMDSv1) and IMDSv2. You can disable IMDSv1. To prevent future nodes and Pods in the node group from using MDSv1, set **DisableIMDSv1** to **true**. For more information about IMDS, see [Configuring the instance metadata service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html).
+ **VpcId**: Select the ID for the [VPC](creating-a-vpc.md) that you created.
+ **NodeSecurityGroups**: Select the security group that was created for your Linux node group when you created your [VPC](creating-a-vpc.md). If your Linux nodes have more than one security group attached to them, specify all of them. This for, for example, if the Linux node group was created with `eksctl`.
+ **Subnets**: Choose the subnets that you created. If you created your VPC using the steps in [Create an Amazon VPC for your Amazon EKS cluster](creating-a-vpc.md), then specify only the private subnets within the VPC for your nodes to launch into.
**Important**
If any of the subnets are public subnets, then they must have the automatic public IP address assignment setting enabled. If the setting isn’t enabled for the public subnet, then any nodes that you deploy to that public subnet won’t be assigned a public IP address and won’t be able to communicate with the cluster or other AWS services. If the subnet was deployed before March 26, 2020 using either of the [Amazon EKS AWS CloudFormation VPC templates](creating-a-vpc.md), or by using `eksctl`, then automatic public IP address assignment is disabled for public subnets. For information about how to enable public IP address assignment for a subnet, see [Modifying the public IPv4 addressing attribute for your subnet](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#subnet-public-ip). If the node is deployed to a private subnet, then it’s able to communicate with the cluster and other AWS services through a NAT gateway.
If the subnets don’t have internet access, then make sure that you’re aware of the considerations and extra steps in [Deploy private clusters with limited internet access](private-clusters.md).
If you select AWS Outposts, Wavelength, or Local Zone subnets, then the subnets must not have been passed in when you created the cluster.
1. Acknowledge that the stack might create IAM resources, and then choose **Create stack**.
1. When your stack has finished creating, select it in the console and choose **Outputs**.
1. Record the **NodeInstanceRole** for the node group that was created. You need this when you configure your Amazon EKS Windows nodes.
**Step 2: Enable nodes to join your cluster**
1. Check to see if you already have an `aws-auth` `ConfigMap`.
```
kubectl describe configmap -n kube-system aws-auth
```
1. If you are shown an `aws-auth` `ConfigMap`, then update it as needed.
1. Open the `ConfigMap` for editing.
```
kubectl edit -n kube-system configmap/aws-auth
```
1. Add new `mapRoles` entries as needed. Set the `rolearn` values to the **NodeInstanceRole** values that you recorded in the previous procedures.
```
[...]
data:
mapRoles: |
- rolearn:
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn:
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- eks:kube-proxy-windows
[...]
```
1. Save the file and exit your text editor.
1. If you received an error stating "`Error from server (NotFound): configmaps "aws-auth" not found`, then apply the stock `ConfigMap`.
1. Download the configuration map.
```
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/aws-auth-cm-windows.yaml
```
1. In the `aws-auth-cm-windows.yaml` file, set the `rolearn` values to the applicable **NodeInstanceRole** values that you recorded in the previous procedures. You can do this with a text editor, or by replacing the example values and running the following command:
```
sed -i.bak -e 's||my-node-linux-instance-role|' \
-e 's||my-node-windows-instance-role|' aws-auth-cm-windows.yaml
```
**Important**
Don’t modify any other lines in this file.
Don’t use the same IAM role for both Windows and Linux nodes.
1. Apply the configuration. This command might take a few minutes to finish.
```
kubectl apply -f aws-auth-cm-windows.yaml
```
1. Watch the status of your nodes and wait for them to reach the `Ready` status.
```
kubectl get nodes --watch
```
Enter `Ctrl`\$1`C` to return to a shell prompt.
**Note**
If you receive any authorization or resource type errors, see [Unauthorized or access denied (`kubectl`)](troubleshooting.md#unauthorized) in the troubleshooting topic.
If nodes fail to join the cluster, then see [Nodes fail to join cluster](troubleshooting.md#worker-node-fail) in the Troubleshooting chapter.
**Step 3: Additional actions**
1. (Optional) Deploy a [sample application](sample-deployment.md) to test your cluster and Windows nodes.
1. (Optional) If the **AmazonEKS\$1CNI\$1Policy** managed IAM policy (if you have an `IPv4` cluster) or the *AmazonEKS\$1CNI\$1IPv6\$1Policy* (that you [created yourself](cni-iam-role.md#cni-iam-role-create-ipv6-policy) if you have an `IPv6` cluster) is attached to your [Amazon EKS node IAM role](create-node-role.md), we recommend assigning it to an IAM role that you associate to the Kubernetes `aws-node` service account instead. For more information, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md).
1. We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
# Create self-managed Ubuntu Linux nodes
**Note**
Managed node groups might offer some advantages for your use case. For more information, see [Simplify node lifecycle with managed node groups](managed-node-groups.md).
This topic describes how to launch Auto Scaling groups of [Ubuntu on Amazon Elastic Kubernetes Service (EKS)](https://cloud-images.ubuntu.com/aws-eks/) or [Ubuntu Pro on Amazon Elastic Kubernetes Service (EKS)](https://ubuntu.com/blog/ubuntu-pro-for-eks-is-now-generally-available) nodes that register with your Amazon EKS cluster. Ubuntu and Ubuntu Pro for EKS are based on the official Ubuntu Minimal LTS, include the custom AWS kernel that is jointly developed with AWS, and have been built specifically for EKS. Ubuntu Pro adds additional security coverage by supporting EKS extended support periods, kernel livepatch, FIPS compliance and the ability to run unlimited Pro containers.
After the nodes join the cluster, you can deploy containerized applications to them. For more information, visit the documentation for [Ubuntu on AWS](https://documentation.ubuntu.com/aws/en/latest/) and [Custom AMI support](https://eksctl.io/usage/custom-ami-support/) in the `eksctl` documentation.
**Important**
Amazon EKS nodes are standard Amazon EC2 instances, and you are billed for them based on normal Amazon EC2 instance prices. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/).
You can launch Ubuntu nodes in Amazon EKS extended clusters on AWS Outposts, but you can’t launch them in local clusters on AWS Outposts. For more information, see [Deploy Amazon EKS on-premises with AWS Outposts](eks-outposts.md).
You can deploy to Amazon EC2 instances with `x86` or Arm processors. However, instances that have Inferentia chips might need to install the [Neuron SDK](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) first.
This procedure requires `eksctl` version `0.215.0` or later. You can check your version with the following command:
```
eksctl version
```
For instructions on how to install or upgrade `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.NOTE: This procedure only works for clusters that were created with `eksctl`.
1. Copy the following contents to your device. Replace `my-cluster` with the name of your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphabetic character and can’t be longer than 100 characters. Replace `ng-ubuntu` with a name for your node group. The node group name can’t be longer than 63 characters. It must start with letter or digit, but can also include hyphens and underscores for the remaining characters. To deploy on Arm instances, replace `m5.large` with an Arm instance type. Replace `my-ec2-keypair-name` with the name of an Amazon EC2 SSH key pair that you can use to connect using SSH into your nodes with after they launch. If you don’t already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see [Amazon EC2 key pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) in the Amazon EC2 User Guide. Replace all remaining example values with your own values. Once you’ve made the replacements, run the modified command to create the `ubuntu.yaml` file.
**Important**
To deploy a node group to AWS Outposts, AWS Wavelength, or AWS Local Zone subnets, don’t pass AWS Outposts, AWS Wavelength, or AWS Local Zone subnets when you create the cluster. You must specify the subnets in the following example. For more information see [Create a nodegroup from a config file](https://eksctl.io/usage/nodegroups/#creating-a-nodegroup-from-a-config-file) and [Config file schema](https://eksctl.io/usage/schema/) in the `eksctl` documentation. Replace *region-code* with the AWS Region that your cluster is in.
```
cat >ubuntu.yaml <
When a new Amazon EKS optimized AMI is released, consider replacing the nodes in your self-managed node group with the new AMI. Likewise, if you have updated the Kubernetes version for your Amazon EKS cluster, update the nodes to use nodes with the same Kubernetes version.
**Important**
This topic covers node updates for self-managed nodes. If you are using [managed node groups](managed-node-groups.md), see [Update a managed node group for your cluster](update-managed-node-group.md).
There are two basic ways to update self-managed node groups in your clusters to use a new AMI:
** [Migrate applications to a new node group](migrate-stack.md) **
Create a new node group and migrate your Pods to that group. Migrating to a new node group is more graceful than simply updating the AMI ID in an existing AWS CloudFormation stack. This is because the migration process [taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) the old node group as `NoSchedule` and drains the nodes after a new stack is ready to accept the existing Pod workload.
** [Update an AWS CloudFormation node stack](update-stack.md) **
Update the AWS CloudFormation stack for an existing node group to use the new AMI. This method isn’t supported for node groups that were created with `eksctl`.
# Migrate applications to a new node group
This topic describes how you can create a new node group, gracefully migrate your existing applications to the new group, and remove the old node group from your cluster. You can migrate to a new node group using `eksctl` or the AWS Management Console.
+ [`eksctl`](#eksctl_migrate_apps)
+ [AWS Management Console and AWS CLI](#console_migrate_apps)
## `eksctl`
**Migrate your applications to a new node group with `eksctl` **
For more information on using eksctl for migration, see [Unmanaged nodegroups](https://eksctl.io/usage/nodegroup-unmanaged/) in the `eksctl` documentation.
This procedure requires `eksctl` version `0.215.0` or later. You can check your version with the following command:
```
eksctl version
```
For instructions on how to install or upgrade `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
**Note**
This procedure only works for clusters and node groups that were created with `eksctl`.
1. Retrieve the name of your existing node groups, replacing *my-cluster* with your cluster name.
```
eksctl get nodegroups --cluster=my-cluster
```
An example output is as follows.
```
CLUSTER NODEGROUP CREATED MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID
default standard-nodes 2019-05-01T22:26:58Z 1 4 3 t3.medium ami-05a71d034119ffc12
```
1. Launch a new node group with `eksctl` with the following command. In the command, replace every *example value* with your own values. The version number can’t be later than the Kubernetes version for your control plane. Also, it can’t be more than two minor versions earlier than the Kubernetes version for your control plane. We recommend that you use the same version as your control plane.
We recommend blocking Pod access to IMDS if the following conditions are true:
+ You plan to assign IAM roles to all of your Kubernetes service accounts so that Pods only have the minimum permissions that they need.
+ No Pods in the cluster require access to the Amazon EC2 instance metadata service (IMDS) for other reasons, such as retrieving the current AWS Region.
For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
To block Pod access to IMDS, add the `--disable-pod-imds` option to the following command.
**Note**
For more available flags and their descriptions, see https://eksctl.io/.
```
eksctl create nodegroup \
--cluster my-cluster \
--version 1.35 \
--name standard-nodes-new \
--node-type t3.medium \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--managed=false
```
1. When the previous command completes, verify that all of your nodes have reached the `Ready` state with the following command:
```
kubectl get nodes
```
1. Delete the original node group with the following command. In the command, replace every *example value* with your cluster and node group names:
```
eksctl delete nodegroup --cluster my-cluster --name standard-nodes-old
```
## AWS Management Console and AWS CLI
**Migrate your applications to a new node group with the AWS Management Console and AWS CLI**
1. Launch a new node group by following the steps that are outlined in [Create self-managed Amazon Linux nodes](launch-workers.md).
1. When your stack has finished creating, select it in the console and choose **Outputs**.
1. Record the **NodeInstanceRole** for the node group that was created. You need this to add the new Amazon EKS nodes to your cluster.
**Note**
If you attached any additional IAM policies to your old node group IAM role, attach those same policies to your new node group IAM role to maintain that functionality on the new group. This applies to you if you added permissions for the [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler), for example.
1. Update the security groups for both node groups so that they can communicate with each other. For more information, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md).
1. Record the security group IDs for both node groups. This is shown as the **NodeSecurityGroup** value in the AWS CloudFormation stack outputs.
You can use the following AWS CLI commands to get the security group IDs from the stack names. In these commands, `oldNodes` is the AWS CloudFormation stack name for your older node stack, and `newNodes` is the name of the stack that you are migrating to. Replace every *example value* with your own values.
```
oldNodes="old_node_CFN_stack_name"
newNodes="new_node_CFN_stack_name"
oldSecGroup=$(aws cloudformation describe-stack-resources --stack-name $oldNodes \
--query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \
--output text)
newSecGroup=$(aws cloudformation describe-stack-resources --stack-name $newNodes \
--query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \
--output text)
```
1. Add ingress rules to each node security group so that they accept traffic from each other.
The following AWS CLI commands add inbound rules to each security group that allow all traffic on all protocols from the other security group. This configuration allows Pods in each node group to communicate with each other while you’re migrating your workload to the new group.
```
aws ec2 authorize-security-group-ingress --group-id $oldSecGroup \
--source-group $newSecGroup --protocol -1
aws ec2 authorize-security-group-ingress --group-id $newSecGroup \
--source-group $oldSecGroup --protocol -1
```
1. Edit the `aws-auth` configmap to map the new node instance role in RBAC.
```
kubectl edit configmap -n kube-system aws-auth
```
Add a new `mapRoles` entry for the new node group.
```
apiVersion: v1
data:
mapRoles: |
- rolearn: ARN of instance role (not instance profile)
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes>
- rolearn: arn:aws:iam::111122223333:role/nodes-1-16-NodeInstanceRole-U11V27W93CX5
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
```
Replace the *ARN of instance role (not instance profile)* snippet with the **NodeInstanceRole** value that you recorded in a [previous step](#node-instance-role-step). Then, save and close the file to apply the updated configmap.
1. Watch the status of your nodes and wait for your new nodes to join your cluster and reach the `Ready` status.
```
kubectl get nodes --watch
```
1. (Optional) If you’re using the [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler), scale the deployment down to zero (0) replicas to avoid conflicting scaling actions.
```
kubectl scale deployments/cluster-autoscaler --replicas=0 -n kube-system
```
1. Use the following command to taint each of the nodes that you want to remove with `NoSchedule`. This is so that new Pods aren’t scheduled or rescheduled on the nodes that you’re replacing. For more information, see [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) in the Kubernetes documentation.
```
kubectl taint nodes node_name key=value:NoSchedule
```
If you’re upgrading your nodes to a new Kubernetes version, you can identify and taint all of the nodes of a particular Kubernetes version (in this case, `1.33`) with the following code snippet. The version number can’t be later than the Kubernetes version of your control plane. It also can’t be more than two minor versions earlier than the Kubernetes version of your control plane. We recommend that you use the same version as your control plane.
```
K8S_VERSION=1.33
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
echo "Tainting $node"
kubectl taint nodes $node key=value:NoSchedule
done
```
1. Determine your cluster’s DNS provider.
```
kubectl get deployments -l k8s-app=kube-dns -n kube-system
```
An example output is as follows. This cluster is using CoreDNS for DNS resolution, but your cluster can return `kube-dns` instead):
```
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
coredns 1 1 1 1 31m
```
1. If your current deployment is running fewer than two replicas, scale out the deployment to two replicas. Replace *coredns* with `kubedns` if your previous command output returned that instead.
```
kubectl scale deployments/coredns --replicas=2 -n kube-system
```
1. Drain each of the nodes that you want to remove from your cluster with the following command:
```
kubectl drain node_name --ignore-daemonsets --delete-local-data
```
If you’re upgrading your nodes to a new Kubernetes version, identify and drain all of the nodes of a particular Kubernetes version (in this case, *1.33*) with the following code snippet.
```
K8S_VERSION=1.33
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
echo "Draining $node"
kubectl drain $node --ignore-daemonsets --delete-local-data
done
```
1. After your old nodes finished draining, revoke the security group inbound rules you authorized earlier. Then, delete the AWS CloudFormation stack to terminate the instances.
**Note**
If you attached any additional IAM policies to your old node group IAM role, such as adding permissions for the [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler), detach those additional policies from the role before you can delete your AWS CloudFormation stack.
1. Revoke the inbound rules that you created for your node security groups earlier. In these commands, `oldNodes` is the AWS CloudFormation stack name for your older node stack, and `newNodes` is the name of the stack that you are migrating to.
```
oldNodes="old_node_CFN_stack_name"
newNodes="new_node_CFN_stack_name"
oldSecGroup=$(aws cloudformation describe-stack-resources --stack-name $oldNodes \
--query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \
--output text)
newSecGroup=$(aws cloudformation describe-stack-resources --stack-name $newNodes \
--query 'StackResources[?ResourceType==`AWS::EC2::SecurityGroup`].PhysicalResourceId' \
--output text)
aws ec2 revoke-security-group-ingress --group-id $oldSecGroup \
--source-group $newSecGroup --protocol -1
aws ec2 revoke-security-group-ingress --group-id $newSecGroup \
--source-group $oldSecGroup --protocol -1
```
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Select your old node stack.
1. Choose **Delete**.
1. In the **Delete stack** confirmation dialog box, choose **Delete stack**.
1. Edit the `aws-auth` configmap to remove the old node instance role from RBAC.
```
kubectl edit configmap -n kube-system aws-auth
```
Delete the `mapRoles` entry for the old node group.
```
apiVersion: v1
data:
mapRoles: |
- rolearn: arn:aws:iam::111122223333:role/nodes-1-16-NodeInstanceRole-W70725MZQFF8
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam::111122223333:role/nodes-1-15-NodeInstanceRole-U11V27W93CX5
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes>
```
Save and close the file to apply the updated configmap.
1. (Optional) If you are using the [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler), scale the deployment back to one replica.
**Note**
You must also tag your new Auto Scaling group appropriately (for example, `k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster`) and update the command for your Cluster Autoscaler deployment to point to the newly tagged Auto Scaling group. For more information, see [Cluster Autoscaler on AWS](https://github.com/kubernetes/autoscaler/tree/cluster-autoscaler-release-1.3/cluster-autoscaler/cloudprovider/aws).
```
kubectl scale deployments/cluster-autoscaler --replicas=1 -n kube-system
```
1. (Optional) Verify that you’re using the latest version of the [Amazon VPC CNI plugin for Kubernetes](https://github.com/aws/amazon-vpc-cni-k8s). You might need to update your CNI version to use the latest supported instance types. For more information, see [Assign IPs to Pods with the Amazon VPC CNI](managing-vpc-cni.md).
1. If your cluster is using `kube-dns` for DNS resolution (see [[migrate-determine-dns-step]](#migrate-determine-dns-step)), scale in the `kube-dns` deployment to one replica.
```
kubectl scale deployments/kube-dns --replicas=1 -n kube-system
```
# Update an AWS CloudFormation node stack
This topic describes how you can update an existing AWS CloudFormation self-managed node stack with a new AMI. You can use this procedure to update your nodes to a new version of Kubernetes following a cluster update. Otherwise, you can update to the latest Amazon EKS optimized AMI for an existing Kubernetes version.
**Important**
This topic covers node updates for self-managed nodes. For information about using [Simplify node lifecycle with managed node groups](managed-node-groups.md), see [Update a managed node group for your cluster](update-managed-node-group.md).
The latest default Amazon EKS node AWS CloudFormation template is configured to launch an instance with the new AMI into your cluster before removing an old one, one at a time. This configuration ensures that you always have your Auto Scaling group’s desired count of active instances in your cluster during the rolling update.
**Note**
This method isn’t supported for node groups that were created with `eksctl`. If you created your cluster or node group with `eksctl`, see [Migrate applications to a new node group](migrate-stack.md).
1. Determine the DNS provider for your cluster.
```
kubectl get deployments -l k8s-app=kube-dns -n kube-system
```
An example output is as follows. This cluster is using CoreDNS for DNS resolution, but your cluster might return `kube-dns` instead. Your output might look different depending on the version of `kubectl` that you’re using.
```
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
coredns 1 1 1 1 31m
```
1. If your current deployment is running fewer than two replicas, scale out the deployment to two replicas. Replace *coredns* with `kube-dns` if your previous command output returned that instead.
```
kubectl scale deployments/coredns --replicas=2 -n kube-system
```
1. (Optional) If you’re using the Kubernetes [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), scale the deployment down to zero (0) replicas to avoid conflicting scaling actions.
```
kubectl scale deployments/cluster-autoscaler --replicas=0 -n kube-system
```
1. Determine the instance type and desired instance count of your current node group. You enter these values later when you update the AWS CloudFormation template for the group.
1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
1. In the left navigation pane, choose **Launch Configurations**, and note the instance type for your existing node launch configuration.
1. In the left navigation pane, choose **Auto Scaling Groups**, and note the **Desired** instance count for your existing node Auto Scaling group.
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Select your node group stack, and then choose **Update**.
1. Select **Replace current template** and select **Amazon S3 URL**.
1. For **Amazon S3 URL**, paste the following URL into the text area to ensure that you’re using the latest version of the node AWS CloudFormation template. Then, choose **Next**:
```
https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2022-12-23/amazon-eks-nodegroup.yaml
```
1. On the **Specify stack details** page, fill out the following parameters, and choose **Next**:
+ **NodeAutoScalingGroupDesiredCapacity** – Enter the desired instance count that you recorded in a [previous step](#existing-worker-settings-step). Or, enter your new desired number of nodes to scale to when your stack is updated.
+ **NodeAutoScalingGroupMaxSize** – Enter the maximum number of nodes to which your node Auto Scaling group can scale out. This value must be at least one node more than your desired capacity. This is so that you can perform a rolling update of your nodes without reducing your node count during the update.
+ **NodeInstanceType** – Choose the instance type your recorded in a [previous step](#existing-worker-settings-step). Alternatively, choose a different instance type for your nodes. Before choosing a different instance type, review [Choose an optimal Amazon EC2 node instance type](choosing-instance-type.md). Each Amazon EC2 instance type supports a maximum number of elastic network interfaces (network interface) and each network interface supports a maximum number of IP addresses. Because each worker node and Pod ,is assigned its own IP address, it’s important to choose an instance type that will support the maximum number of Pods that you want to run on each Amazon EC2 node. For a list of the number of network interfaces and IP addresses supported by instance types, see [IP addresses per network interface per instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI). For example, the `m5.large` instance type supports a maximum of 30 IP addresses for the worker node and Pods.
**Note**
The supported instance types for the latest version of the [Amazon VPC CNI plugin for Kubernetes](https://github.com/aws/amazon-vpc-cni-k8s) are shown in [vpc\$1ip\$1resource\$1limit.go](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/pkg/vpc/vpc_ip_resource_limit.go) on GitHub. You might need to update your Amazon VPC CNI plugin for Kubernetes version to use the latest supported instance types. For more information, see [Assign IPs to Pods with the Amazon VPC CNI](managing-vpc-cni.md).
**Important**
Some instance types might not be available in all AWS Regions.
+ **NodeImageIdSSMParam** – The Amazon EC2 Systems Manager parameter of the AMI ID that you want to update to. The following value uses the latest Amazon EKS optimized AMI for Kubernetes version `1.35`.
```
/aws/service/eks/optimized-ami/1.35/amazon-linux-2/recommended/image_id
```
You can replace *1.35* with a [platform-version](https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html) that’s the same. Or, it should be up to one version earlier than the Kubernetes version running on your control plane. We recommend that you keep your nodes at the same version as your control plane. You can also replace *amazon-linux-2* with a different AMI type. For more information, see [Retrieve recommended Amazon Linux AMI IDs](retrieve-ami-id.md).
**Note**
Using the Amazon EC2 Systems Manager parameter enables you to update your nodes in the future without having to look up and specify an AMI ID. If your AWS CloudFormation stack is using this value, any stack update always launches the latest recommended Amazon EKS optimized AMI for your specified Kubernetes version. This is even the case even if you don’t change any values in the template.
+ **NodeImageId** – To use your own custom AMI, enter the ID for the AMI to use.
**Important**
This value overrides any value specified for **NodeImageIdSSMParam**. If you want to use the **NodeImageIdSSMParam** value, ensure that the value for **NodeImageId** is blank.
+ **DisableIMDSv1** – By default, each node supports the Instance Metadata Service Version 1 (IMDSv1) and IMDSv2. However, you can disable IMDSv1. Select **true** if you don’t want any nodes or any Pods scheduled in the node group to use IMDSv1. For more information about IMDS, see [Configuring the instance metadata service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html). If you’ve implemented IAM roles for service accounts, assign necessary permissions directly to all Pods that require access to AWS services. This way, no Pods in your cluster require access to IMDS for other reasons, such as retrieving the current AWS Region. Then, you can also disable access to IMDSv2 for Pods that don’t use host networking. For more information, see [Restrict access to the instance profile assigned to the worker node](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node).
1. (Optional) On the **Options** page, tag your stack resources. Choose **Next**.
1. On the **Review** page, review your information, acknowledge that the stack might create IAM resources, and then choose **Update stack**.
**Note**
The update of each node in the cluster takes several minutes. Wait for the update of all nodes to complete before performing the next steps.
1. If your cluster’s DNS provider is `kube-dns`, scale in the `kube-dns` deployment to one replica.
```
kubectl scale deployments/kube-dns --replicas=1 -n kube-system
```
1. (Optional) If you are using the Kubernetes [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), scale the deployment back to your desired amount of replicas.
```
kubectl scale deployments/cluster-autoscaler --replicas=1 -n kube-system
```
1. (Optional) Verify that you’re using the latest version of the [Amazon VPC CNI plugin for Kubernetes](https://github.com/aws/amazon-vpc-cni-k8s). You might need to update your Amazon VPC CNI plugin for Kubernetes version to use the latest supported instance types. For more information, see [Assign IPs to Pods with the Amazon VPC CNI](managing-vpc-cni.md).
# Simplify compute management with AWS Fargate
This topic discusses using Amazon EKS to run Kubernetes Pods on AWS Fargate. Fargate is a technology that provides on-demand, right-sized compute capacity for [containers](https://aws.amazon.com/what-are-containers). With Fargate, you don’t have to provision, configure, or scale groups of virtual machines on your own to run containers. You also don’t need to choose server types, decide when to scale your node groups, or optimize cluster packing.
You can control which Pods start on Fargate and how they run with [Fargate profiles](fargate-profile.md). Fargate profiles are defined as part of your Amazon EKS cluster. Amazon EKS integrates Kubernetes with Fargate by using controllers that are built by AWS using the upstream, extensible model provided by Kubernetes. These controllers run as part of the Amazon EKS managed Kubernetes control plane and are responsible for scheduling native Kubernetes Pods onto Fargate. The Fargate controllers include a new scheduler that runs alongside the default Kubernetes scheduler in addition to several mutating and validating admission controllers. When you start a Pod that meets the criteria for running on Fargate, the Fargate controllers that are running in the cluster recognize, update, and schedule the Pod onto Fargate.
This topic describes the different components of Pods that run on Fargate, and calls out special considerations for using Fargate with Amazon EKS.
## AWS Fargate considerations
Here are some things to consider about using Fargate on Amazon EKS.
+ Each Pod that runs on Fargate has its own compute boundary. They don’t share the underlying kernel, CPU resources, memory resources, or elastic network interface with another Pod.
+ Network Load Balancers and Application Load Balancers (ALBs) can be used with Fargate with IP targets only. For more information, see [Create a network load balancer](network-load-balancing.md#network-load-balancer) and [Route application and HTTP traffic with Application Load Balancers](alb-ingress.md).
+ Fargate exposed services only run on target type IP mode, and not on node IP mode. The recommended way to check the connectivity from a service running on a managed node and a service running on Fargate is to connect via service name.
+ Pods must match a Fargate profile at the time that they’re scheduled to run on Fargate. Pods that don’t match a Fargate profile might be stuck as `Pending`. If a matching Fargate profile exists, you can delete pending Pods that you have created to reschedule them onto Fargate.
+ Daemonsets aren’t supported on Fargate. If your application requires a daemon, reconfigure that daemon to run as a sidecar container in your Pods.
+ Privileged containers aren’t supported on Fargate.
+ Pods running on Fargate can’t specify `HostPort` or `HostNetwork` in the Pod manifest.
+ The default `nofile` and `nproc` soft limit is 1024 and the hard limit is 65535 for Fargate Pods.
+ GPUs aren’t currently available on Fargate.
+ Pods that run on Fargate are only supported on private subnets (with NAT gateway access to AWS services, but not a direct route to an Internet Gateway), so your cluster’s VPC must have private subnets available. For clusters without outbound internet access, see [Deploy private clusters with limited internet access](private-clusters.md).
+ You can use the [Adjust pod resources with Vertical Pod Autoscaler](vertical-pod-autoscaler.md) to set the initial correct size of CPU and memory for your Fargate Pods, and then use the [Scale pod deployments with Horizontal Pod Autoscaler](horizontal-pod-autoscaler.md) to scale those Pods. If you want the Vertical Pod Autoscaler to automatically re-deploy Pods to Fargate with larger CPU and memory combinations, set the mode for the Vertical Pod Autoscaler to either `Auto` or `Recreate` to ensure correct functionality. For more information, see the [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#quick-start) documentation on GitHub.
+ DNS resolution and DNS hostnames must be enabled for your VPC. For more information, see [Viewing and updating DNS support for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating).
+ Amazon EKS Fargate adds defense-in-depth for Kubernetes applications by isolating each Pod within a Virtual Machine (VM). This VM boundary prevents access to host-based resources used by other Pods in the event of a container escape, which is a common method of attacking containerized applications and gain access to resources outside of the container.
Using Amazon EKS doesn’t change your responsibilities under the [shared responsibility model](security.md). You should carefully consider the configuration of cluster security and governance controls. The safest way to isolate an application is always to run it in a separate cluster.
+ Fargate profiles support specifying subnets from VPC secondary CIDR blocks. You might want to specify a secondary CIDR block. This is because there’s a limited number of IP addresses available in a subnet. As a result, there’s also a limited number of Pods that can be created in the cluster. By using different subnets for Pods, you can increase the number of available IP addresses. For more information, see [Adding IPv4 CIDR blocks to a VPC.](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html#vpc-resize)
+ The Amazon EC2 instance metadata service (IMDS) isn’t available to Pods that are deployed to Fargate nodes. If you have Pods that are deployed to Fargate that need IAM credentials, assign them to your Pods using [IAM roles for service accounts](iam-roles-for-service-accounts.md). If your Pods need access to other information available through IMDS, then you must hard code this information into your Pod spec. This includes the AWS Region or Availability Zone that a Pod is deployed to.
+ You can’t deploy Fargate Pods to AWS Outposts, AWS Wavelength, or AWS Local Zones.
+ Amazon EKS must periodically patch Fargate Pods to keep them secure. We attempt the updates in a way that reduces impact, but there are times when Pods must be deleted if they aren’t successfully evicted. There are some actions you can take to minimize disruption. For more information, see [Set actions for AWS Fargate OS patching events](fargate-pod-patching.md).
+ The [Amazon VPC CNI plugin for Amazon EKS](https://github.com/aws/amazon-vpc-cni-plugins) is installed on Fargate nodes. You can’t use [Alternate CNI plugins for Amazon EKS clusters](alternate-cni-plugins.md) with Fargate nodes.
+ A Pod running on Fargate automatically mounts an Amazon EFS file system, without needing manual driver installation steps. You can’t use dynamic persistent volume provisioning with Fargate nodes, but you can use static provisioning.
+ Amazon EKS doesn’t support Fargate Spot.
+ You can’t mount Amazon EBS volumes to Fargate Pods.
+ You can run the Amazon EBS CSI controller on Fargate nodes, but the Amazon EBS CSI node DaemonSet can only run on Amazon EC2 instances.
+ After a [Kubernetes Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) is marked `Completed` or `Failed`, the Pods that the Job creates normally continue to exist. This behavior allows you to view your logs and results, but with Fargate you will incur costs if you don’t clean up the Job afterwards.
To automatically delete the related Pods after a Job completes or fails, you can specify a time period using the time-to-live (TTL) controller. The following example shows specifying `.spec.ttlSecondsAfterFinished` in your Job manifest.
```
apiVersion: batch/v1
kind: Job
metadata:
name: busybox
spec:
template:
spec:
containers:
- name: busybox
image: busybox
command: ["/bin/sh", "-c", "sleep 10"]
restartPolicy: Never
ttlSecondsAfterFinished: 60 # <-- TTL controller
```
## Fargate Comparison Table
| Criteria | AWS Fargate |
| --- | --- |
| Can be deployed to [AWS Outposts](https://docs.aws.amazon.com/outposts/latest/userguide/what-is-outposts.html) | No |
| Can be deployed to an [AWS Local Zone](local-zones.md) | No |
| Can run containers that require Windows | No |
| Can run containers that require Linux | Yes |
| Can run workloads that require the Inferentia chip | No |
| Can run workloads that require a GPU | No |
| Can run workloads that require Arm processors | No |
| Can run AWS [Bottlerocket](https://aws.amazon.com/bottlerocket/) | No |
| Pods share a kernel runtime environment with other Pods | No – Each Pod has a dedicated kernel |
| Pods share CPU, memory, storage, and network resources with other Pods. | No – Each Pod has dedicated resources and can be sized independently to maximize resource utilization. |
| Pods can use more hardware and memory than requested in Pod specs | No – The Pod can be re-deployed using a larger vCPU and memory configuration though. |
| Must deploy and manage Amazon EC2 instances | No |
| Must secure, maintain, and patch the operating system of Amazon EC2 instances | No |
| Can provide bootstrap arguments at deployment of a node, such as extra [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) arguments. | No |
| Can assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node. | No |
| Can SSH into node | No – There’s no node host operating system to SSH to. |
| Can deploy your own custom AMI to nodes | No |
| Can deploy your own custom CNI to nodes | No |
| Must update node AMI on your own | No |
| Must update node Kubernetes version on your own | No – You don’t manage nodes. |
| Can use Amazon EBS storage with Pods | No |
| Can use Amazon EFS storage with Pods | [Yes](efs-csi.md) |
| Can use Amazon FSx for Lustre storage with Pods | No |
| Can use Network Load Balancer for services | Yes, when using the [Create a network load balancer](network-load-balancing.md#network-load-balancer) |
| Pods can run in a public subnet | No |
| Can assign different VPC security groups to individual Pods | Yes |
| Can run Kubernetes DaemonSets | No |
| Support `HostPort` and `HostNetwork` in the Pod manifest | No |
| AWS Region availability | [Some Amazon EKS supported regions](https://docs.aws.amazon.com/general/latest/gr/eks.html) |
| Can run containers on Amazon EC2 dedicated hosts | No |
| Pricing | Cost of an individual Fargate memory and CPU configuration. Each Pod has its own cost. For more information, see [AWS Fargate pricing](https://aws.amazon.com/fargate/pricing/). |
# Get started with AWS Fargate for your cluster
This topic describes how to get started running Pods on AWS Fargate with your Amazon EKS cluster.
If you restrict access to the public endpoint of your cluster using CIDR blocks, we recommend that you also enable private endpoint access. This way, Fargate Pods can communicate with the cluster. Without the private endpoint enabled, the CIDR blocks that you specify for public access must include the outbound sources from your VPC. For more information, see [Cluster API server endpoint](cluster-endpoint.md).
**Prerequisite**
An existing cluster. If you don’t already have an Amazon EKS cluster, see [Get started with Amazon EKS](getting-started.md).
## Step 1: Ensure that existing nodes can communicate with Fargate Pods
If you’re working with a new cluster with no nodes, or a cluster with only managed node groups (see [Simplify node lifecycle with managed node groups](managed-node-groups.md)), you can skip to [Step 2: Create a Fargate Pod execution role](#fargate-sg-pod-execution-role).
Assume that you’re working with an existing cluster that already has nodes that are associated with it. Make sure that Pods on these nodes can communicate freely with the Pods that are running on Fargate. Pods that are running on Fargate are automatically configured to use the cluster security group for the cluster that they’re associated with. Ensure that any existing nodes in your cluster can send and receive traffic to and from the cluster security group. Managed node groups are automatically configured to use the cluster security group as well, so you don’t need to modify or check them for this compatibility (see [Simplify node lifecycle with managed node groups](managed-node-groups.md)).
For existing node groups that were created with `eksctl` or the Amazon EKS managed AWS CloudFormation templates, you can add the cluster security group to the nodes manually. Or, alternatively, you can modify the Auto Scaling group launch template for the node group to attach the cluster security group to the instances. For more information, see [Changing an instance’s security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#SG_Changing_Group_Membership) in the *Amazon VPC User Guide*.
You can check for a security group for your cluster in the AWS Management Console under the **Networking** section for the cluster. Or, you can do this using the following AWS CLI command. When using this command, replace `` with the name of your cluster.
```
aws eks describe-cluster --name --query cluster.resourcesVpcConfig.clusterSecurityGroupId
```
## Step 2: Create a Fargate Pod execution role
When your cluster creates Pods on AWS Fargate, the components that run on the Fargate infrastructure must make calls to AWS APIs on your behalf. The Amazon EKS Pod execution role provides the IAM permissions to do this. To create an AWS Fargate Pod execution role, see [Amazon EKS Pod execution IAM role](pod-execution-role.md).
**Note**
If you created your cluster with `eksctl` using the `--fargate` option, your cluster already has a Pod execution role that you can find in the IAM console with the pattern `eksctl-my-cluster-FargatePodExecutionRole-ABCDEFGHIJKL`. Similarly, if you use `eksctl` to create your Fargate profiles, `eksctl` creates your Pod execution role if one isn’t already created.
## Step 3: Create a Fargate profile for your cluster
Before you can schedule Pods that are running on Fargate in your cluster, you must define a Fargate profile that specifies which Pods use Fargate when they’re launched. For more information, see [Define which Pods use AWS Fargate when launched](fargate-profile.md).
**Note**
If you created your cluster with `eksctl` using the `--fargate` option, then a Fargate profile is already created for your cluster with selectors for all Pods in the `kube-system` and `default` namespaces. Use the following procedure to create Fargate profiles for any other namespaces you would like to use with Fargate.
You can create a Fargate profile using either of these tools:
+ [`eksctl`](#eksctl_fargate_profile_create)
+ [AWS Management Console](#console_fargate_profile_create)
### `eksctl`
This procedure requires `eksctl` version `0.215.0` or later. You can check your version with the following command:
```
eksctl version
```
For instructions on how to install or upgrade `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
**To create a Fargate profile with `eksctl` **
Create your Fargate profile with the following `eksctl` command, replacing every `` with your own values. You’re required to specify a namespace. However, the `--labels` option isn’t required.
```
eksctl create fargateprofile \
--cluster \
--name \
--namespace \
--labels
```
You can use certain wildcards for `` and `` labels. For more information, see [Fargate profile wildcards](fargate-profile.md#fargate-profile-wildcards).
### AWS Management Console
**To create a Fargate profile with AWS Management Console **
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the cluster to create a Fargate profile for.
1. Choose the **Compute** tab.
1. Under **Fargate profiles**, choose **Add Fargate profile**.
1. On the **Configure Fargate profile** page, do the following:
1. For **Name**, enter a name for your Fargate profile. The name must be unique.
1. For **Pod execution role**, choose the Pod execution role to use with your Fargate profile. Only the IAM roles with the `eks-fargate-pods.amazonaws.com` service principal are shown. If you don’t see any roles listed, you must create one. For more information, see [Amazon EKS Pod execution IAM role](pod-execution-role.md).
1. Modify the selected **Subnets** as needed.
**Note**
Only private subnets are supported for Pods that are running on Fargate.
1. For **Tags**, you can optionally tag your Fargate profile. These tags don’t propagate to other resources that are associated with the profile such as Pods.
1. Choose **Next**.
1. On the **Configure Pod selection** page, do the following:
1. For **Namespace**, enter a namespace to match for Pods.
+ You can use specific namespaces to match, such as `kube-system` or `default`.
+ You can use certain wildcards (for example, `prod-*`) to match multiple namespaces (for example, `prod-deployment` and `prod-test`). For more information, see [Fargate profile wildcards](fargate-profile.md#fargate-profile-wildcards).
1. (Optional) Add Kubernetes labels to the selector. Specifically add them to the one that the Pods in the specified namespace need to match.
+ You can add the label `infrastructure: fargate` to the selector so that only Pods in the specified namespace that also have the `infrastructure: fargate` Kubernetes label match the selector.
+ You can use certain wildcards (for example, `key?: value?`) to match multiple namespaces (for example, `keya: valuea` and `keyb: valueb`). For more information, see [Fargate profile wildcards](fargate-profile.md#fargate-profile-wildcards).
1. Choose **Next**.
1. On the **Review and create** page, review the information for your Fargate profile and choose **Create**.
## Step 4: Update CoreDNS
By default, CoreDNS is configured to run on Amazon EC2 infrastructure on Amazon EKS clusters. If you want to *only* run your Pods on Fargate in your cluster, complete the following steps.
**Note**
If you created your cluster with `eksctl` using the `--fargate` option, then you can skip to [Next steps](#fargate-gs-next-steps).
1. Create a Fargate profile for CoreDNS with the following command. Replace `` with your cluster name, `<111122223333>` with your account ID, `` with the name of your Pod execution role, and `<000000000000000a>`, `<000000000000000b>`, and `<000000000000000c>` with the IDs of your private subnets. If you don’t have a Pod execution role, you must create one first (see [Step 2: Create a Fargate Pod execution role](#fargate-sg-pod-execution-role)).
**Important**
The role ARN can’t include a [path](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-friendly-names) other than `/`. For example, if the name of your role is `development/apps/AmazonEKSFargatePodExecutionRole`, you need to change it to `AmazonEKSFargatePodExecutionRole` when specifying the ARN for the role. The format of the role ARN must be ` arn:aws:iam::<111122223333>:role/`.
```
aws eks create-fargate-profile \
--fargate-profile-name coredns \
--cluster-name \
--pod-execution-role-arn arn:aws:iam::<111122223333>:role/ \
--selectors namespace=kube-system,labels={k8s-app=kube-dns} \
--subnets subnet-<000000000000000a> subnet-<000000000000000b> subnet-<000000000000000c>
```
1. Trigger a rollout of the `coredns` deployment.
```
kubectl rollout restart -n kube-system deployment coredns
```
## Next steps
+ You can start migrating your existing applications to run on Fargate with the following workflow.
1. [Create a Fargate profile](fargate-profile.md#create-fargate-profile) that matches your application’s Kubernetes namespace and Kubernetes labels.
1. Delete and re-create any existing Pods so that they’re scheduled on Fargate. Modify the `` and `` to update your specific Pods.
```
kubectl rollout restart -n deployment
```
+ Deploy the [Route application and HTTP traffic with Application Load Balancers](alb-ingress.md) to allow Ingress objects for your Pods running on Fargate.
+ You can use the [Adjust pod resources with Vertical Pod Autoscaler](vertical-pod-autoscaler.md) to set the initial correct size of CPU and memory for your Fargate Pods, and then use the [Scale pod deployments with Horizontal Pod Autoscaler](horizontal-pod-autoscaler.md) to scale those Pods. If you want the Vertical Pod Autoscaler to automatically re-deploy Pods to Fargate with higher CPU and memory combinations, set the Vertical Pod Autoscaler’s mode to either `Auto` or `Recreate`. This is to ensure correct functionality. For more information, see the [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#quick-start) documentation on GitHub.
+ You can set up the [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel) (ADOT) collector for application monitoring by following [these instructions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-otel.html).
# Define which Pods use AWS Fargate when launched
Before you schedule Pods on Fargate in your cluster, you must define at least one Fargate profile that specifies which Pods use Fargate when launched.
As an administrator, you can use a Fargate profile to declare which Pods run on Fargate. You can do this through the profile’s selectors. You can add up to five selectors to each profile. Each selector must contain a namespace. The selector can also include labels. The label field consists of multiple optional key-value pairs. Pods that match a selector are scheduled on Fargate. Pods are matched using a namespace and the labels that are specified in the selector. If a namespace selector is defined without labels, Amazon EKS attempts to schedule all the Pods that run in that namespace onto Fargate using the profile. If a to-be-scheduled Pod matches any of the selectors in the Fargate profile, then that Pod is scheduled on Fargate.
If a Pod matches multiple Fargate profiles, you can specify which profile a Pod uses by adding the following Kubernetes label to the Pod specification: `eks.amazonaws.com/fargate-profile: my-fargate-profile`. The Pod must match a selector in that profile to be scheduled onto Fargate. Kubernetes affinity/anti-affinity rules do not apply and aren’t necessary with Amazon EKS Fargate Pods.
When you create a Fargate profile, you must specify a Pod execution role. This execution role is for the Amazon EKS components that run on the Fargate infrastructure using the profile. It’s added to the cluster’s Kubernetes [Role Based Access Control](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) (RBAC) for authorization. That way, the `kubelet` that runs on the Fargate infrastructure can register with your Amazon EKS cluster and appear in your cluster as a node. The Pod execution role also provides IAM permissions to the Fargate infrastructure to allow read access to Amazon ECR image repositories. For more information, see [Amazon EKS Pod execution IAM role](pod-execution-role.md).
Fargate profiles can’t be changed. However, you can create a new updated profile to replace an existing profile, and then delete the original.
**Note**
Any Pods that are running using a Fargate profile are stopped and put into a pending state when the profile is deleted.
If any Fargate profiles in a cluster are in the `DELETING` status, you must wait until after the Fargate profile is deleted before you create other profiles in that cluster.
**Note**
Fargate does not currently support Kubernetes [topologySpreadConstraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/).
Amazon EKS and Fargate spread Pods across each of the subnets that’s defined in the Fargate profile. However, you might end up with an uneven spread. If you must have an even spread, use two Fargate profiles. Even spread is important in scenarios where you want to deploy two replicas and don’t want any downtime. We recommend that each profile has only one subnet.
## Fargate profile components
The following components are contained in a Fargate profile.
**Pod execution role**
When your cluster creates Pods on AWS Fargate, the `kubelet` that’s running on the Fargate infrastructure must make calls to AWS APIs on your behalf. For example, it needs to make calls to pull container images from Amazon ECR. The Amazon EKS Pod execution role provides the IAM permissions to do this.
When you create a Fargate profile, you must specify a Pod execution role to use with your Pods. This role is added to the cluster’s Kubernetes [Role-based access control](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) (RBAC) for authorization. This is so that the `kubelet` that’s running on the Fargate infrastructure can register with your Amazon EKS cluster and appear in your cluster as a node. For more information, see [Amazon EKS Pod execution IAM role](pod-execution-role.md).
**Subnets**
The IDs of subnets to launch Pods into that use this profile. At this time, Pods that are running on Fargate aren’t assigned public IP addresses. Therefore, only private subnets with no direct route to an Internet Gateway are accepted for this parameter.
**Selectors**
The selectors to match for Pods to use this Fargate profile. You might specify up to five selectors in a Fargate profile. The selectors have the following components:
+ **Namespace** – You must specify a namespace for a selector. The selector only matches Pods that are created in this namespace. However, you can create multiple selectors to target multiple namespaces.
+ **Labels** – You can optionally specify Kubernetes labels to match for the selector. The selector only matches Pods that have all of the labels that are specified in the selector.
## Fargate profile wildcards
In addition to characters allowed by Kubernetes, you’re allowed to use `*` and `?` in the selector criteria for namespaces, label keys, and label values:
+ `*` represents none, one, or multiple characters. For example, `prod*` can represent `prod` and `prod-metrics`.
+ `?` represents a single character (for example, `value?` can represent `valuea`). However, it can’t represent `value` and `value-a`, because `?` can only represent exactly one character.
These wildcard characters can be used in any position and in combination (for example, `prod*`, `*dev`, and `frontend*?`). Other wildcards and forms of pattern matching, such as regular expressions, aren’t supported.
If there are multiple matching profiles for the namespace and labels in the Pod spec, Fargate picks up the profile based on alphanumeric sorting by profile name. For example, if both profile A (with the name `beta-workload`) and profile B (with the name `prod-workload`) have matching selectors for the Pods to be launched, Fargate picks profile A (`beta-workload`) for the Pods. The Pods have labels with profile A on the Pods (for example, `eks.amazonaws.com/fargate-profile=beta-workload`).
If you want to migrate existing Fargate Pods to new profiles that use wildcards, there are two ways to do so:
+ Create a new profile with matching selectors, then delete the old profiles. Pods labeled with old profiles are rescheduled to new matching profiles.
+ If you want to migrate workloads but aren’t sure what Fargate labels are on each Fargate Pod, you can use the following method. Create a new profile with a name that sorts alphanumerically first among the profiles on the same cluster. Then, recycle the Fargate Pods that need to be migrated to new profiles.
## Create a Fargate profile
This section describes how to create a Fargate profile. You also must have created a Pod execution role to use for your Fargate profile. For more information, see [Amazon EKS Pod execution IAM role](pod-execution-role.md). Pods that are running on Fargate are only supported on private subnets with [NAT gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) access to AWS services, but not a direct route to an Internet Gateway. This is so that your cluster’s VPC must have private subnets available.
You can create a profile with the following:
+ [`eksctl`](#eksctl_create_a_fargate_profile)
+ [AWS Management Console](#console_create_a_fargate_profile)
## `eksctl`
**To create a Fargate profile with `eksctl` **
Create your Fargate profile with the following `eksctl` command, replacing every example value with your own values. You’re required to specify a namespace. However, the `--labels` option isn’t required.
```
eksctl create fargateprofile \
--cluster my-cluster \
--name my-fargate-profile \
--namespace my-kubernetes-namespace \
--labels key=value
```
You can use certain wildcards for `my-kubernetes-namespace` and `key=value` labels. For more information, see [Fargate profile wildcards](#fargate-profile-wildcards).
## AWS Management Console
**To create a Fargate profile with AWS Management Console **
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the cluster to create a Fargate profile for.
1. Choose the **Compute** tab.
1. Under **Fargate profiles**, choose **Add Fargate profile**.
1. On the **Configure Fargate profile** page, do the following:
1. For **Name**, enter a unique name for your Fargate profile, such as `my-profile`.
1. For **Pod execution role**, choose the Pod execution role to use with your Fargate profile. Only the IAM roles with the `eks-fargate-pods.amazonaws.com` service principal are shown. If you don’t see any roles listed, you must create one. For more information, see [Amazon EKS Pod execution IAM role](pod-execution-role.md).
1. Modify the selected **Subnets** as needed.
**Note**
Only private subnets are supported for Pods that are running on Fargate.
1. For **Tags**, you can optionally tag your Fargate profile. These tags don’t propagate to other resources that are associated with the profile, such as Pods.
1. Choose **Next**.
1. On the **Configure Pod selection** page, do the following:
1. For **Namespace**, enter a namespace to match for Pods.
+ You can use specific namespaces to match, such as `kube-system` or `default`.
+ You can use certain wildcards (for example, `prod-*`) to match multiple namespaces (for example, `prod-deployment` and `prod-test`). For more information, see [Fargate profile wildcards](#fargate-profile-wildcards).
1. (Optional) Add Kubernetes labels to the selector. Specifically, add them to the one that the Pods in the specified namespace need to match.
+ You can add the label `infrastructure: fargate` to the selector so that only Pods in the specified namespace that also have the `infrastructure: fargate` Kubernetes label match the selector.
+ You can use certain wildcards (for example, `key?: value?`) to match multiple namespaces (for example, `keya: valuea` and `keyb: valueb`). For more information, see [Fargate profile wildcards](#fargate-profile-wildcards).
1. Choose **Next**.
1. On the **Review and create** page, review the information for your Fargate profile and choose **Create**.
# Delete a Fargate profile
This topic describes how to delete a Fargate profile. When you delete a Fargate profile, any Pods that were scheduled onto Fargate with the profile are deleted. If those Pods match another Fargate profile, then they’re scheduled on Fargate with that profile. If they no longer match any Fargate profiles, then they aren’t scheduled onto Fargate and might remain as pending.
Only one Fargate profile in a cluster can be in the `DELETING` status at a time. Wait for a Fargate profile to finish deleting before you can delete any other profiles in that cluster.
You can delete a profile with any of the following tools:
+ [`eksctl`](#eksctl_delete_a_fargate_profile)
+ [AWS Management Console](#console_delete_a_fargate_profile)
+ [AWS CLI](#awscli_delete_a_fargate_profile)
## `eksctl`
**Delete a Fargate profile with `eksctl` **
Use the following command to delete a profile from a cluster. Replace every *example value* with your own values.
```
eksctl delete fargateprofile --name my-profile --cluster my-cluster
```
## AWS Management Console
**Delete a Fargate profile with AWS Management Console **
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. In the left navigation pane, choose **Clusters**. In the list of clusters, choose the cluster that you want to delete the Fargate profile from.
1. Choose the **Compute** tab.
1. Choose the Fargate profile to delete, and then choose **Delete**.
1. On the **Delete Fargate profile** page, enter the name of the profile, and then choose **Delete**.
## AWS CLI
**Delete a Fargate profile with AWS CLI**
Use the following command to delete a profile from a cluster. Replace every *example value* with your own values.
```
aws eks delete-fargate-profile --fargate-profile-name my-profile --cluster-name my-cluster
```
# Understand Fargate Pod configuration details
This section describes some of the unique Pod configuration details for running Kubernetes Pods on AWS Fargate.
## Pod CPU and memory
With Kubernetes, you can define requests, a minimum vCPU amount, and memory resources that are allocated to each container in a Pod. Pods are scheduled by Kubernetes to ensure that at least the requested resources for each Pod are available on the compute resource. For more information, see [Managing compute resources for containers](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) in the Kubernetes documentation.
**Note**
Since Amazon EKS Fargate runs only one Pod per node, the scenario of evicting Pods in case of fewer resources doesn’t occur. All Amazon EKS Fargate Pods run with guaranteed priority, so the requested CPU and memory must be equal to the limit for all of the containers. For more information, see [Configure Quality of Service for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/) in the Kubernetes documentation.
When Pods are scheduled on Fargate, the vCPU and memory reservations within the Pod specification determine how much CPU and memory to provision for the Pod.
+ The maximum request out of any Init containers is used to determine the Init request vCPU and memory requirements.
+ Requests for all long-running containers are added up to determine the long-running request vCPU and memory requirements.
+ The larger of the previous two values is chosen for the vCPU and memory request to use for your Pod.
+ Fargate adds 256 MB to each Pod’s memory reservation for the required Kubernetes components (`kubelet`, `kube-proxy`, and `containerd`).
Fargate rounds up to the following compute configuration that most closely matches the sum of vCPU and memory requests in order to ensure Pods always have the resources that they need to run.
If you don’t specify a vCPU and memory combination, then the smallest available combination is used (.25 vCPU and 0.5 GB memory).
The following table shows the vCPU and memory combinations that are available for Pods running on Fargate.
| vCPU value | Memory value |
| --- | --- |
| .25 vCPU | 0.5 GB, 1 GB, 2 GB |
| .5 vCPU | 1 GB, 2 GB, 3 GB, 4 GB |
| 1 vCPU | 2 GB, 3 GB, 4 GB, 5 GB, 6 GB, 7 GB, 8 GB |
| 2 vCPU | Between 4 GB and 16 GB in 1-GB increments |
| 4 vCPU | Between 8 GB and 30 GB in 1-GB increments |
| 8 vCPU | Between 16 GB and 60 GB in 4-GB increments |
| 16 vCPU | Between 32 GB and 120 GB in 8-GB increments |
The additional memory reserved for the Kubernetes components can cause a Fargate task with more vCPUs than requested to be provisioned. For example, a request for 1 vCPU and 8 GB memory will have 256 MB added to its memory request, and will provision a Fargate task with 2 vCPUs and 9 GB memory, since no task with 1 vCPU and 9 GB memory is available.
There is no correlation between the size of the Pod running on Fargate and the node size reported by Kubernetes with `kubectl get nodes`. The reported node size is often larger than the Pod’s capacity. You can verify Pod capacity with the following command. Replace *default* with your Pod’s namespace and *pod-name* with the name of your Pod.
```
kubectl describe pod --namespace default pod-name
```
An example output is as follows.
```
[...]
annotations:
CapacityProvisioned: 0.25vCPU 0.5GB
[...]
```
The `CapacityProvisioned` annotation represents the enforced Pod capacity and it determines the cost of your Pod running on Fargate. For pricing information for the compute configurations, see [AWS Fargate Pricing](https://aws.amazon.com/fargate/pricing/).
## Fargate storage
A Pod running on Fargate automatically mounts an Amazon EFS file system, without needing manual driver installation steps. You can’t use dynamic persistent volume provisioning with Fargate nodes, but you can use static provisioning. For more information, see [Amazon EFS CSI Driver](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/docs/README.md) on GitHub.
When provisioned, each Pod running on Fargate receives a default 20 GiB of ephemeral storage. This type of storage is deleted after a Pod stops. New Pods launched onto Fargate have encryption of the ephemeral storage volume enabled by default. The ephemeral Pod storage is encrypted with an AES-256 encryption algorithm using AWS Fargate managed keys.
**Note**
The default usable storage for Amazon EKS Pods that run on Fargate is less than 20 GiB. This is because some space is used by the `kubelet` and other Kubernetes modules that are loaded inside the Pod.
You can increase the total amount of ephemeral storage up to a maximum of 175 GiB. To configure the size with Kubernetes, specify the requests of `ephemeral-storage` resource to each container in a Pod. When Kubernetes schedules Pods, it ensures that the sum of the resource requests for each Pod is less than the capacity of the Fargate task. For more information, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) in the Kubernetes documentation.
Amazon EKS Fargate provisions more ephemeral storage than requested for the purposes of system use. For example, a request of 100 GiB will provision a Fargate task with 115 GiB ephemeral storage.
# Set actions for AWS Fargate OS patching events
Amazon EKS periodically patches the OS for AWS Fargate nodes to keep them secure. As part of the patching process, we recycle the nodes to install OS patches. Updates are attempted in a way that creates the least impact on your services. However, if Pods aren’t successfully evicted, there are times when they must be deleted. The following are actions that you can take to minimize potential disruptions:
+ Set appropriate Pod disruption budgets (PDBs) to control the number of Pods that are down simultaneously.
+ Create Amazon EventBridge rules to handle failed evictions before the Pods are deleted.
+ Manually restart your affected pods before the eviction date posted in the notification you receive.
+ Create a notification configuration in AWS User Notifications.
Amazon EKS works closely with the Kubernetes community to make bug fixes and security patches available as quickly as possible. All Fargate Pods start on the most recent Kubernetes patch version, which is available from Amazon EKS for the Kubernetes version of your cluster. If you have a Pod with an older patch version, Amazon EKS might recycle it to update it to the latest version. This ensures that your Pods are equipped with the latest security updates. That way, if there’s a critical [Common Vulnerabilities and Exposures](https://cve.mitre.org/) (CVE) issue, you’re kept up to date to reduce security risks.
When the AWS Fargate OS is updated, Amazon EKS will send you a notification that includes your affected resources and the date of upcoming pod evictions. If the provided eviction date is inconvenient, you have the option to manually restart your affected pods before the eviction date posted in the notification. Any pods created before the time at which you receive the notification are subject to eviction. Refer to the [Kubernetes Documentation](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart) for further instructions on how to manually restart your pods.
To limit the number of Pods that are down at one time when Pods are recycled, you can set Pod disruption budgets (PDBs). You can use PDBs to define minimum availability based on the requirements of each of your applications while still allowing updates to occur. Your PDB’s minimum availability must be less than 100%. For more information, see [Specifying a Disruption Budget for your Application](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) in the Kubernetes Documentation.
Amazon EKS uses the [Eviction API](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#eviction-api) to safely drain the Pod while respecting the PDBs that you set for the application. Pods are evicted by Availability Zone to minimize impact. If the eviction succeeds, the new Pod gets the latest patch and no further action is required.
When the eviction for a Pod fails, Amazon EKS sends an event to your account with details about the Pods that failed eviction. You can act on the message before the scheduled termination time. The specific time varies based on the urgency of the patch. When it’s time, Amazon EKS attempts to evict the Pods again. However, this time a new event isn’t sent if the eviction fails. If the eviction fails again, your existing Pods are deleted periodically so that the new Pods can have the latest patch.
The following is a sample event received when the Pod eviction fails. It contains details about the cluster, Pod name, Pod namespace, Fargate profile, and the scheduled termination time.
```
{
"version": "0",
"id": "12345678-90ab-cdef-0123-4567890abcde",
"detail-type": "EKS Fargate Pod Scheduled Termination",
"source": "aws.eks",
"account": "111122223333",
"time": "2021-06-27T12:52:44Z",
"region": "region-code",
"resources": [
"default/my-database-deployment"
],
"detail": {
"clusterName": "my-cluster",
"fargateProfileName": "my-fargate-profile",
"podName": "my-pod-name",
"podNamespace": "default",
"evictErrorMessage": "Cannot evict pod as it would violate the pod's disruption budget",
"scheduledTerminationTime": "2021-06-30T12:52:44.832Z[UTC]"
}
}
```
In addition, having multiple PDBs associated with a Pod can cause an eviction failure event. This event returns the following error message.
```
"evictErrorMessage": "This pod has multiple PodDisruptionBudget, which the eviction subresource does not support",
```
You can create a desired action based on this event. For example, you can adjust your Pod disruption budget (PDB) to control how the Pods are evicted. More specifically, suppose that you start with a PDB that specifies the target percentage of Pods that are available. Before your Pods are force terminated during an upgrade, you can adjust the PDB to a different percentage of Pods. To receive this event, you must create an Amazon EventBridge rule in the AWS account and AWS Region that the cluster belongs to. The rule must use the following **Custom pattern**. For more information, see [Creating Amazon EventBridge rules that react to events](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in the *Amazon EventBridge User Guide*.
```
{
"source": ["aws.eks"],
"detail-type": ["EKS Fargate Pod Scheduled Termination"]
}
```
A suitable target can be set for the event to capture it. For a complete list of available targets, see [Amazon EventBridge targets](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html) in the *Amazon EventBridge User Guide*. You can also create a notification configuration in AWS User Notifications. When using the AWS Management Console to create the notification, under **Event Rules**, choose **Elastic Kubernetes Service (EKS)** for ** AWS service name** and **EKS Fargate Pod Scheduled Termination** for **Event type**. For more information, see [Getting started with AWS User Notifications](https://docs.aws.amazon.com/notifications/latest/userguide/getting-started.html) in the AWS User Notifications User Guide.
See [FAQs: Fargate Pod eviction notice](https://repost.aws/knowledge-center/fargate-pod-eviction-notice) in * AWS re:Post* for frequently asked questions regarding EKS Pod Evictions.
# Collect AWS Fargate app and usage metrics
You can collect system metrics and CloudWatch usage metrics for AWS Fargate.
## Application metrics
For applications running on Amazon EKS and AWS Fargate, you can use the AWS Distro for OpenTelemetry (ADOT). ADOT allows you to collect system metrics and send them to CloudWatch Container Insights dashboards. To get started with ADOT for applications running on Fargate, see [Using CloudWatch Container Insights with AWS Distro for OpenTelemetry](https://aws-otel.github.io/docs/getting-started/container-insights) in the ADOT documentation.
## Usage metrics
You can use CloudWatch usage metrics to provide visibility into your account’s usage of resources. Use these metrics to visualize your current service usage on CloudWatch graphs and dashboards.
AWS Fargate usage metrics correspond to AWS service quotas. You can configure alarms that alert you when your usage approaches a service quota. For more information about Fargate service quotas, see [View and manage Amazon EKS and Fargate service quotas](service-quotas.md).
AWS Fargate publishes the following metrics in the ` AWS/Usage` namespace.
| Metric | Description |
| --- | --- |
| `ResourceCount` | The total number of the specified resource running on your account. The resource is defined by the dimensions associated with the metric. |
The following dimensions are used to refine the usage metrics that are published by AWS Fargate.
| Dimension | Description |
| --- | --- |
| `Service` | The name of the AWS service containing the resource. For AWS Fargate usage metrics, the value for this dimension is `Fargate`. |
| `Type` | The type of entity that’s being reported. Currently, the only valid value for AWS Fargate usage metrics is `Resource`. |
| `Resource` | The type of resource that’s running. Currently, AWS Fargate returns information on your Fargate On-Demand usage. The resource value for Fargate On-Demand usage is `OnDemand`. [NOTE] ==== Fargate On-Demand usage combines Amazon EKS Pods using Fargate, Amazon ECS tasks using the Fargate launch type and Amazon ECS tasks using the `FARGATE` capacity provider. ==== |
| `Class` | The class of resource being tracked. Currently, AWS Fargate doesn’t use the class dimension. |
### Creating a CloudWatch alarm to monitor Fargate resource usage metrics
AWS Fargate provides CloudWatch usage metrics that correspond to the AWS service quotas for Fargate On-Demand resource usage. In the Service Quotas console, you can visualize your usage on a graph. You can also configure alarms that alert you when your usage approaches a service quota. For more information, see [Collect AWS Fargate app and usage metrics](#monitoring-fargate-usage).
Use the following steps to create a CloudWatch alarm based on the Fargate resource usage metrics.
1. Open the Service Quotas console at https://console.aws.amazon.com/servicequotas/.
1. In the left navigation pane, choose ** AWS services**.
1. From the ** AWS services** list, search for and select ** AWS Fargate**.
1. In the **Service quotas** list, choose the Fargate usage quota you want to create an alarm for.
1. In the Amazon CloudWatch alarms section, choose **Create**.
1. For **Alarm threshold**, choose the percentage of your applied quota value that you want to set as the alarm value.
1. For **Alarm name**, enter a name for the alarm and then choose **Create**.
# Start AWS Fargate logging for your cluster
Amazon EKS on Fargate offers a built-in log router based on Fluent Bit. This means that you don’t explicitly run a Fluent Bit container as a sidecar, but Amazon runs it for you. All that you have to do is configure the log router. The configuration happens through a dedicated `ConfigMap` that must meet the following criteria:
+ Named `aws-logging`
+ Created in a dedicated namespace called `aws-observability`
+ Can’t exceed 5300 characters.
Once you’ve created the `ConfigMap`, Amazon EKS on Fargate automatically detects it and configures the log router with it. Fargate uses a version of AWS for Fluent Bit, an upstream compliant distribution of Fluent Bit managed by AWS. For more information, see [AWS for Fluent Bit](https://github.com/aws/aws-for-fluent-bit) on GitHub.
The log router allows you to use the breadth of services at AWS for log analytics and storage. You can stream logs from Fargate directly to Amazon CloudWatch, Amazon OpenSearch Service. You can also stream logs to destinations such as [Amazon S3](https://aws.amazon.com/s3/), [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/), and partner tools through [Amazon Data Firehose](https://aws.amazon.com/kinesis/data-firehose/).
+ An existing Fargate profile that specifies an existing Kubernetes namespace that you deploy Fargate Pods to. For more information, see [Step 3: Create a Fargate profile for your cluster](fargate-getting-started.md#fargate-gs-create-profile).
+ An existing Fargate Pod execution role. For more information, see [Step 2: Create a Fargate Pod execution role](fargate-getting-started.md#fargate-sg-pod-execution-role).
## Log router configuration
**Important**
For logs to be successfully published, there must be network access from the VPC that your cluster is in to the log destination. This mainly concerns users customizing egress rules for their VPC. For an example using CloudWatch, see [Using CloudWatch Logs with interface VPC endpoints](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch-logs-and-interface-VPC.html) in the *Amazon CloudWatch Logs User Guide*.
In the following steps, replace every *example value* with your own values.
1. Create a dedicated Kubernetes namespace named `aws-observability`.
1. Save the following contents to a file named `aws-observability-namespace.yaml` on your computer. The value for `name` must be `aws-observability` and the `aws-observability: enabled` label is required.
```
kind: Namespace
apiVersion: v1
metadata:
name: aws-observability
labels:
aws-observability: enabled
```
1. Create the namespace.
```
kubectl apply -f aws-observability-namespace.yaml
```
1. Create a `ConfigMap` with a `Fluent Conf` data value to ship container logs to a destination. Fluent Conf is Fluent Bit, which is a fast and lightweight log processor configuration language that’s used to route container logs to a log destination of your choice. For more information, see [Configuration File](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file) in the Fluent Bit documentation.
**Important**
The main sections included in a typical `Fluent Conf` are `Service`, `Input`, `Filter`, and `Output`. The Fargate log router however, only accepts:
The `Filter` and `Output` sections.
A `Parser` section.
If you provide any other sections, they will be rejected.
The Fargate log router manages the `Service` and `Input` sections. It has the following `Input` section, which can’t be modified and isn’t needed in your `ConfigMap`. However, you can get insights from it, such as the memory buffer limit and the tag applied for logs.
```
[INPUT]
Name tail
Buffer_Max_Size 66KB
DB /var/log/flb_kube.db
Mem_Buf_Limit 45MB
Path /var/log/containers/*.log
Read_From_Head On
Refresh_Interval 10
Rotate_Wait 30
Skip_Long_Lines On
Tag kube.*
```
When creating the `ConfigMap`, take into account the following rules that Fargate uses to validate fields:
+ `[FILTER]`, `[OUTPUT]`, and `[PARSER]` are supposed to be specified under each corresponding key. For example, `[FILTER]` must be under `filters.conf`. You can have one or more `[FILTER]`s under `filters.conf`. The `[OUTPUT]` and `[PARSER]` sections should also be under their corresponding keys. By specifying multiple `[OUTPUT]` sections, you can route your logs to different destinations at the same time.
+ Fargate validates the required keys for each section. `Name` and `match` are required for each `[FILTER]` and `[OUTPUT]`. `Name` and `format` are required for each `[PARSER]`. The keys are case-insensitive.
+ Environment variables such as `${ENV_VAR}` aren’t allowed in the `ConfigMap`.
+ The indentation has to be the same for either directive or key-value pair within each `filters.conf`, `output.conf`, and `parsers.conf`. Key-value pairs have to be indented more than directives.
+ Fargate validates against the following supported filters: `grep`, `parser`, `record_modifier`, `rewrite_tag`, `throttle`, `nest`, `modify`, and `kubernetes`.
+ Fargate validates against the following supported output: `es`, `firehose`, `kinesis_firehose`, `cloudwatch`, `cloudwatch_logs`, and `kinesis`.
+ At least one supported `Output` plugin has to be provided in the `ConfigMap` to enable logging. `Filter` and `Parser` aren’t required to enable logging.
You can also run Fluent Bit on Amazon EC2 using the desired configuration to troubleshoot any issues that arise from validation. Create your `ConfigMap` using one of the following examples.
**Important**
Amazon EKS Fargate logging doesn’t support dynamic configuration of a `ConfigMap`. Any changes to a `ConfigMap` are applied to new Pods only. Changes aren’t applied to existing Pods.
Create a `ConfigMap` using the example for your desired log destination.
**Note**
You can also use Amazon Kinesis Data Streams for your log destination. If you use Kinesis Data Streams, make sure that the pod execution role has been granted the `kinesis:PutRecords` permission. For more information, see Amazon Kinesis Data Streams [Permissions](https://docs.fluentbit.io/manual/pipeline/outputs/kinesis#permissions) in the *Fluent Bit: Official Manual*.
**Example**
------
#### [ CloudWatch ]
You have two output options when using CloudWatch:
+ [An output plugin written in C](https://docs.fluentbit.io/manual/v/1.5/pipeline/outputs/cloudwatch)
+ [An output plugin written in Golang](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit)
The following example shows you how to use the `cloudwatch_logs` plugin to send logs to CloudWatch.
1. Save the following contents to a file named `aws-logging-cloudwatch-configmap.yaml`. Replace *region-code* with the AWS Region that your cluster is in. The parameters under `[OUTPUT]` are required.
```
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: "false" # Set to true to ship Fluent Bit process logs to CloudWatch.
filters.conf: |
[FILTER]
Name parser
Match *
Key_name log
Parser crio
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Buffer_Size 0
Kube_Meta_Cache_TTL 300s
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match kube.*
region region-code
log_group_name my-logs
log_stream_prefix from-fluent-bit-
log_retention_days 60
auto_create_group true
parsers.conf: |
[PARSER]
Name crio
Format Regex
Regex ^(?