**Help improve this page**
To contribute to this user guide, choose the **Edit this page on GitHub** link that is located in the right pane of every page.
# Amazon EKS cluster lifecycle and configuration
This section provides in-depth guidance on the full lifecycle management of Kubernetes clusters and different ways to configure them. It covers cluster creation, monitoring cluster health, updating Kubernetes versions, and deleting clusters. It also includes advanced topics on how to configure endpoint access, enable/disable Windows support, set up private clusters, implement autoscaling, and how to enhance resilience with zonal shift for traffic redirection in multi-AZ setups.
**Topics**
+ [
# Create an Amazon EKS Auto Mode cluster
](create-cluster-auto.md)
+ [
# Create an Amazon EKS cluster
](create-cluster.md)
+ [
# Amazon EKS Provisioned Control Plane
](eks-provisioned-control-plane.md)
+ [
# Prepare for Kubernetes version upgrades and troubleshoot misconfigurations with cluster insights
](cluster-insights.md)
+ [
# Update existing cluster to new Kubernetes version
](update-cluster.md)
+ [
# Delete a cluster
](delete-cluster.md)
+ [
# Cluster API server endpoint
](cluster-endpoint.md)
+ [
# Deploy Windows nodes on EKS clusters
](windows-support.md)
+ [
# Disable Windows support
](disable-windows-support.md)
+ [
# Deploy private clusters with limited internet access
](private-clusters.md)
+ [
# Scale cluster compute with Karpenter and Cluster Autoscaler
](autoscaling.md)
+ [
# Learn about Amazon Application Recovery Controller (ARC) zonal shift in Amazon EKS
](zone-shift.md)
+ [
# Enable EKS zonal shift to avoid impaired Availability Zones
](zone-shift-enable.md)
# Create an Amazon EKS Auto Mode cluster
This topic provides detailed instructions for creating an Amazon EKS Auto Mode cluster using advanced configuration options. It covers prerequisites, networking options, and add-on configurations. The process includes setting up IAM roles, configuring cluster settings, specifying networking parameters, and selecting add-ons. Users can create clusters using either the AWS Management Console or the AWS CLI, with step-by-step guidance provided for both methods.
For users seeking a less complex setup process, refer to the following for simplified cluster creation steps:
+ [Create an EKS Auto Mode Cluster with the eksctl CLI](automode-get-started-eksctl.md)
+ [Create an EKS Auto Mode Cluster with the AWS CLI](automode-get-started-cli.md)
+ [Create an EKS Auto Mode Cluster with the AWS Management Console](automode-get-started-console.md)
This advanced configuration guide is intended for users who require more granular control over their EKS Auto Mode cluster setup and are familiar with Amazon EKS concepts and requirements. Before proceeding with the advanced configuration, ensure you have met all prerequisites and have a thorough understanding of the networking and IAM requirements for EKS Auto Mode clusters.
EKS Auto Mode requires additional IAM permissions. For more information, see:
+ [IAM Roles for EKS Auto Mode Clusters](automode-get-started-cli.md#auto-mode-create-roles)
+ [Learn about identity and access in EKS Auto Mode](auto-learn-iam.md)
**Note**
If you want to create a cluster without EKS Auto Mode, see [Create an Amazon EKS cluster](create-cluster.md).
This topic covers advanced configuration. If you are looking to get started with EKS Auto Mode, see [Create a cluster with Amazon EKS Auto Mode](create-auto.md).
## Prerequisites
+ An existing VPC and subnets that meet [Amazon EKS requirements](network-reqs.md). Before you deploy a cluster for production use, we recommend that you have a thorough understanding of the VPC and subnet requirements. If you don’t have a VPC and subnets, you can create them using an [Amazon EKS provided AWS CloudFormation template](creating-a-vpc.md).
+ The `kubectl` command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is `1.29`, you can use `kubectl` version `1.28`, `1.29`, or `1.30` with it. To install or upgrade `kubectl`, see [Set up `kubectl` and `eksctl`](install-kubectl.md).
+ Version `2.12.3` or later or version `1.27.160` or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use `aws --version`. To install the latest version, see [Installing](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) and [Quick configuration with aws configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config) in the * AWS Command Line Interface User Guide*.
+ An [IAM principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-principal) with permissions to create and modify EKS and IAM resources.
## Create cluster - AWS console
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose **Add cluster** and then choose **Create**.
1. Under *Configuration options*, select **Custom configuration**.
+ This topic covers custom configuration. For information about Quick configuration, see [Create an EKS Auto Mode Cluster with the AWS Management Console](automode-get-started-console.md).
1. Confirm **Use EKS Auto Mode** is enabled.
+ This topic covers creating clusters with EKS Auto Mode. For more information about creating clusters without EKS Auto Mode, see [Create an Amazon EKS cluster](create-cluster.md).
1. On the **Configure cluster** page, enter the following fields:
+ **Name** – A name for your cluster. The name can contain only alphanumeric characters (case-sensitive), hyphens, and underscores. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
+ **Cluster IAM role** – Choose the Amazon EKS cluster IAM role that you created to allow the Kubernetes control plane to manage AWS resources on your behalf. If you haven’t previously created a Cluster IAM role for EKS Auto Mode, select the **Create recommended role** button to create the role with the required permissions in the IAM console.
+ **Kubernetes version** – The version of Kubernetes to use for your cluster. We recommend selecting the latest version, unless you need an earlier version.
+ **Upgrade policy** — The Kubernetes version policy you would like to set for your cluster. If you want your cluster to only run on a standard support version, you can choose **Standard**. If you want your cluster to enter extended support at the end of standard support for a version, you can choose **Extended**. If you select a Kubernetes version that is currently in extended support, you can not select standard support as an option.
1. In the **Auto Mode Compute** section of the configure cluster page, enter the following fields:
+ **Node pools** — Determine if you want to use the build in node pools. For more information, see [Enable or Disable Built-in NodePools](set-builtin-node-pools.md).
+ **Node IAM role** — If you enable any of the built-in node pools, you need to select a Node IAM Role. EKS Auto Mode will assign this role to new nodes. You cannot change this value after the cluster is created. If you haven’t previously created a Node IAM role for EKS Auto Mode, select the Create recommended role button to create the role with the required permissions. For more information about this role, see [Learn about identity and access in EKS Auto Mode](auto-learn-iam.md).
1. In the **Cluster access** section of the configure cluster page, enter the following fields:
+ **Bootstrap cluster administrator access** — The cluster creator is automatically a Kubernetes administrator. If you want to disable this, select **Disallow cluster administrator access**.
+ **Cluster authentication mode** — EKS Auto Mode requires EKS access entries, the EKS API authentication mode. You can optionally enable the `ConfigMap` authentication mode by selecting **EKS API and ConfigMap**.
1. Enter the remaining fields on the configure cluster page:
+ **Secrets encryption** – (Optional) Choose to enable secrets encryption of Kubernetes secrets using a KMS key. You can also enable this after you create your cluster. Before you enable this capability, make sure that you’re familiar with the information in [Encrypt Kubernetes secrets with KMS on existing clusters](enable-kms.md).
+ **ARC Zonal shift** — EKS Auto Mode does not support Arc Zonal shift.
+ **Tags** – (Optional) Add any tags to your cluster. For more information, see [Organize Amazon EKS resources with tags](eks-using-tags.md).
When you’re done with this page, choose **Next**.
1. On the **Specify networking** page, select values for the following fields:
+ **VPC** – Choose an existing VPC that meets [Amazon EKS VPC requirements](network-reqs.md#network-requirements-vpc) to create your cluster in. Before choosing a VPC, we recommend that you’re familiar with all of the requirements and considerations in [View Amazon EKS networking requirements for VPC and subnets](network-reqs.md). You can’t change which VPC you want to use after cluster creation. If no VPCs are listed, then you need to create one first. For more information, see [Create an Amazon VPC for your Amazon EKS cluster](creating-a-vpc.md).
+ **Subnets** – By default, all available subnets in the VPC specified in the previous field are preselected. You must select at least two.
The subnets that you choose must meet the [Amazon EKS subnet requirements](network-reqs.md#network-requirements-subnets). Before selecting subnets, we recommend that you’re familiar with all of the [Amazon EKS VPC and subnet requirements and considerations](network-reqs.md).
**Security groups** – (Optional) Specify one or more security groups that you want Amazon EKS to associate to the network interfaces that it creates.
Whether you choose any security groups or not, Amazon EKS creates a security group that enables communication between your cluster and your VPC. Amazon EKS associates this security group, and any that you choose, to the network interfaces that it creates. For more information about the cluster security group that Amazon EKS creates, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md). You can modify the rules in the cluster security group that Amazon EKS creates.
+ **Choose cluster IP address family** – You can choose either **IPv4** and **IPv6**.
Kubernetes assigns `IPv4` addresses to Pods and services, by default. Before deciding to use the `IPv6` family, make sure that you’re familiar with all of the considerations and requirements in the [VPC requirements and considerations](network-reqs.md#network-requirements-vpc), [Subnet requirements and considerations](network-reqs.md#network-requirements-subnets), [View Amazon EKS security group requirements for clusters](sec-group-reqs.md), and [Learn about IPv6 addresses to clusters, Pods, and services](cni-ipv6.md) topics. If you choose the `IPv6` family, you can’t specify an address range for Kubernetes to assign `IPv6` service addresses from like you can for the `IPv4` family. Kubernetes assigns service addresses from the unique local address range (`fc00::/7`).
+ (Optional) Choose **Configure Kubernetes Service IP address range** and specify a **Service `IPv4` range**.
Specifying your own range can help prevent conflicts between Kubernetes services and other networks peered or connected to your VPC. Enter a range in CIDR notation. For example: `10.2.0.0/16`.
The CIDR block must meet the following requirements:
+ Be within one of the following ranges: `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`.
+ Have a minimum size of `/24` and a maximum size of `/12`.
+ Not overlap with the range of the VPC for your Amazon EKS resources.
You can only specify this option when using the `IPv4` address family and only at cluster creation. If you don’t specify this, then Kubernetes assigns service IP addresses from either the `10.100.0.0/16` or `172.20.0.0/16` CIDR blocks.
+ For **Cluster endpoint access**, select an option. After your cluster is created, you can change this option. Before selecting a non-default option, make sure to familiarize yourself with the options and their implications. For more information, see [Cluster API server endpoint](cluster-endpoint.md).
When you’re done with this page, choose **Next**.
1. (Optional) On the **Configure observability** page, choose which **Metrics** and **Control plane logging** options to turn on. By default, each log type is turned off.
+ For more information about the Prometheus metrics option, see [Step 1: Turn on Prometheus metrics](prometheus.md#turn-on-prometheus-metrics).
+ For more information about the **Control plane logging** options, see [Send control plane logs to CloudWatch Logs](control-plane-logs.md).
+ When you’re done with this page, choose **Next**.
1. On the **Select add-ons** page, choose the add-ons that you want to add to your cluster. You can choose as many **Amazon EKS add-ons** and ** AWS Marketplace add-ons** as you require. If the ** AWS Marketplace add-ons** that you want to install isn’t listed, you can click the page numbering to view additional page results or search for available ** AWS Marketplace add-ons** by entering text in the search box. You can also filter by **category**, **vendor**, or **pricing model** and then choose the add-ons from the search results. When creating a cluster, you can view, select, and install any add-on that supports EKS Pod Identities as detailed in [Learn how EKS Pod Identity grants pods access to AWS services](pod-identities.md).
+ EKS Auto Mode automates the functionality of certain add-ons. If you plan to deploy EKS Managed Node Groups to your EKS Auto Mode Cluster, select **Additional Amazon EKS Add-ons** and review the options. You may need to install add-ons such as CoreDNS and kube-proxy. EKS will only install the add-ons in this section on self-managed nodes and node groups.
+ When you’re done with this page, choose **Next**.
1. On the **Configure selected add-ons settings** page, select the version that you want to install. You can always update to a later version after cluster creation.
For add-ons that support EKS Pod Identities, you can use the console to automatically generate the role with the name, AWS managed policy, and trust policy prepopulated specifically for the add-on. You can re-use existing roles or create new roles for supported add-ons. For the steps to use the console to create roles for add-ons that support EKS Pod Identities, see [Create add-on (AWS Console)](creating-an-add-on.md#create_add_on_console). If an add-on does not support EKS Pod Identity, a message displays with instructions to use the wizard to create the IAM roles for service accounts (IRSA) after the cluster is created.
You can update the configuration of each add-on after cluster creation. For more information about configuring add-ons, see [Update an Amazon EKS add-on](updating-an-add-on.md). When you’re done with this page, choose **Next**.
1. On the **Review and create** page, review the information that you entered or selected on the previous pages. If you need to make changes, choose **Edit**. When you’re satisfied, choose **Create**. The **Status** field shows **CREATING** while the cluster is provisioned.
**Note**
You might receive an error that one of the Availability Zones in your request doesn’t have sufficient capacity to create an Amazon EKS cluster. If this happens, the error output contains the Availability Zones that can support a new cluster. Retry creating your cluster with at least two subnets that are located in the supported Availability Zones for your account. For more information, see [Insufficient capacity](troubleshooting.md#ice).
Cluster provisioning takes several minutes.
## Create cluster - AWS CLI
The following CLI instructions cover creating IAM resources and creating the cluster.
### Create an EKS Auto Mode Cluster IAM Role
#### Step 1: Create the Trust Policy
Create a trust policy that allows the Amazon EKS service to assume the role. Save the policy as `trust-policy.json`:
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "eks.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
}
```
#### Step 2: Create the IAM Role
Use the trust policy to create the Cluster IAM Role:
```
aws iam create-role \
--role-name AmazonEKSAutoClusterRole \
--assume-role-policy-document file://trust-policy.json
```
#### Step 3: Note the Role ARN
Retrieve and save the ARN of the new role for use in subsequent steps:
```
aws iam get-role --role-name AmazonEKSAutoClusterRole --query "Role.Arn" --output text
```
#### Step 4: Attach Required Policies
Attach the following AWS managed policies to the Cluster IAM Role to grant the necessary permissions:
**AmazonEKSClusterPolicy**:
```
aws iam attach-role-policy \
--role-name AmazonEKSAutoClusterRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
```
**AmazonEKSComputePolicy**:
```
aws iam attach-role-policy \
--role-name AmazonEKSAutoClusterRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSComputePolicy
```
**AmazonEKSBlockStoragePolicy**:
```
aws iam attach-role-policy \
--role-name AmazonEKSAutoClusterRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy
```
**AmazonEKSLoadBalancingPolicy**:
```
aws iam attach-role-policy \
--role-name AmazonEKSAutoClusterRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy
```
**AmazonEKSNetworkingPolicy**:
```
aws iam attach-role-policy \
--role-name AmazonEKSAutoClusterRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy
```
### Create an EKS Auto Mode Node IAM Role
#### Step 1: Create the Trust Policy
Create a trust policy that allows the Amazon EKS service to assume the role. Save the policy as `node-trust-policy.json`:
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
#### Step 2: Create the Node IAM Role
Use the **node-trust-policy.json** file from the previous step to define which entities can assume the role. Run the following command to create the Node IAM Role:
```
aws iam create-role \
--role-name AmazonEKSAutoNodeRole \
--assume-role-policy-document file://node-trust-policy.json
```
#### Step 3: Note the Role ARN
After creating the role, retrieve and save the ARN of the Node IAM Role. You will need this ARN in subsequent steps. Use the following command to get the ARN:
```
aws iam get-role --role-name AmazonEKSAutoNodeRole --query "Role.Arn" --output text
```
#### Step 4: Attach Required Policies
Attach the following AWS managed policies to the Node IAM Role to provide the necessary permissions:
**AmazonEKSWorkerNodeMinimalPolicy**:
```
aws iam attach-role-policy \
--role-name AmazonEKSAutoNodeRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy
```
**AmazonEC2ContainerRegistryPullOnly**:
```
aws iam attach-role-policy \
--role-name AmazonEKSAutoNodeRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly
```
### Create cluster
1. Create your cluster with the command that follows. Before running the command, make the following replacements:
+ Replace *region-code* with the AWS Region that you want to create your cluster in.
+ Replace *my-cluster* with a name for your cluster. The name can contain only alphanumeric characters (case-sensitive), hyphens, and underscores. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
+ Replace *1.30* with any [Amazon EKS supported version](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
+ Replace *111122223333* with your account ID
+ If you have created differently named IAM Roles for the Cluster and Node roles, replace the ARNs.
+ Replace the values for `subnetIds` with your own. You can also add additional IDs. You must specify at least two subnet IDs.
The subnets that you choose must meet the [Amazon EKS subnet requirements](network-reqs.md#network-requirements-subnets). Before selecting subnets, we recommend that you’re familiar with all of the [Amazon EKS VPC and subnet requirements and considerations](network-reqs.md).
+ If you don’t want to specify a security group ID, remove `,securityGroupIds=sg-` from the command. If you want to specify one or more security group IDs, replace the values for `securityGroupIds` with your own. You can also add additional IDs.
Whether you choose any security groups or not, Amazon EKS creates a security group that enables communication between your cluster and your VPC. Amazon EKS associates this security group, and any that you choose, to the network interfaces that it creates. For more information about the cluster security group that Amazon EKS creates, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md). You can modify the rules in the cluster security group that Amazon EKS creates.
```
aws eks create-cluster \
--region region-code \
--name my-cluster \
--kubernetes-version 1.30 \
--role-arn arn:aws:iam::111122223333:role/AmazonEKSAutoClusterRole \
--resources-vpc-config '{"subnetIds": ["subnet-ExampleID1","subnet-ExampleID2"], "securityGroupIds": ["sg-ExampleID1"], "endpointPublicAccess": true, "endpointPrivateAccess": true}' \
--compute-config '{"enabled": true, "nodeRoleArn": "arn:aws:iam::111122223333:role/AmazonEKSAutoNodeRole", "nodePools": ["general-purpose", "system"]}' \
--kubernetes-network-config '{"elasticLoadBalancing": {"enabled": true}}' \
--storage-config '{"blockStorage": {"enabled": true}}' \
--access-config '{"authenticationMode": "API"}'
```
**Note**
You might receive an error that one of the Availability Zones in your request doesn’t have sufficient capacity to create an Amazon EKS cluster. If this happens, the error output contains the Availability Zones that can support a new cluster. Retry creating your cluster with at least two subnets that are located in the supported Availability Zones for your account. For more information, see [Insufficient capacity](troubleshooting.md#ice).
The following are optional settings that, if required, must be added to the previous command. You can only enable these options when you create the cluster, not after.
+ If you want to specify which `IPv4` Classless Inter-domain Routing (CIDR) block Kubernetes assigns service IP addresses from, you must specify it by adding the `--kubernetes-network-config serviceIpv4Cidr=` to the following command.
Specifying your own range can help prevent conflicts between Kubernetes services and other networks peered or connected to your VPC. Enter a range in CIDR notation. For example: `10.2.0.0/16`.
The CIDR block must meet the following requirements:
+ Be within one of the following ranges: `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`.
+ Have a minimum size of `/24` and a maximum size of `/12`.
+ Not overlap with the range of the VPC for your Amazon EKS resources.
You can only specify this option when using the `IPv4` address family and only at cluster creation. If you don’t specify this, then Kubernetes assigns service IP addresses from either the `10.100.0.0/16` or `172.20.0.0/16` CIDR blocks.
+ If you’re creating a cluster and want the cluster to assign `IPv6` addresses to Pods and services instead of `IPv4` addresses, add `--kubernetes-network-config ipFamily=ipv6` to the following command.
Kubernetes assigns `IPv4` addresses to Pods and services, by default. Before deciding to use the `IPv6` family, make sure that you’re familiar with all of the considerations and requirements in the [VPC requirements and considerations](network-reqs.md#network-requirements-vpc), [Subnet requirements and considerations](network-reqs.md#network-requirements-subnets), [View Amazon EKS security group requirements for clusters](sec-group-reqs.md), and [Learn about IPv6 addresses to clusters, Pods, and services](cni-ipv6.md) topics. If you choose the `IPv6` family, you can’t specify an address range for Kubernetes to assign `IPv6` service addresses from like you can for the `IPv4` family. Kubernetes assigns service addresses from the unique local address range (`fc00::/7`).
1. It takes several minutes to provision the cluster. You can query the status of your cluster with the following command.
```
aws eks describe-cluster --region region-code --name my-cluster --query "cluster.status"
```
## Next steps
+ [Connect kubectl to an EKS cluster by creating a kubeconfig file](create-kubeconfig.md)
+ [Grant IAM users access to Kubernetes with EKS access entries](access-entries.md)
+ [Enable secrets encryption for your cluster](enable-kms.md).
+ [Configure logging for your cluster](control-plane-logs.md).
+ [Add nodes to your cluster](eks-compute.md).
# Create an Amazon EKS cluster
**Note**
This topic covers creating Amazon EKS clusters without EKS Auto Mode.
For detailed instructions on creating an EKS Auto Mode cluster, see [Create an Amazon EKS Auto Mode cluster](create-cluster-auto.md).
To get started with EKS Auto Mode, see [Get started with Amazon EKS – EKS Auto Mode](getting-started-automode.md).
This topic provides an overview of the available options and describes what to consider when you create an Amazon EKS cluster. If you need to create a cluster with your on-premises infrastructure as the compute for nodes, see [Create an Amazon EKS cluster with hybrid nodes](hybrid-nodes-cluster-create.md). If this is your first time creating an Amazon EKS cluster, we recommend that you follow one of our guides in [Get started with Amazon EKS](getting-started.md). These guides help you to create a simple, default cluster without expanding into all of the available options.
## Prerequisites
+ An existing VPC and subnets that meet [Amazon EKS requirements](network-reqs.md). Before you deploy a cluster for production use, we recommend that you have a thorough understanding of the VPC and subnet requirements. If you don’t have a VPC and subnets, you can create them using an [Amazon EKS provided AWS CloudFormation template](creating-a-vpc.md).
+ The `kubectl` command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. To install or upgrade `kubectl`, see [Set up `kubectl` and `eksctl`](install-kubectl.md).
+ Version `2.12.3` or later or version `1.27.160` or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use `aws --version | cut -d / -f2 | cut -d ' ' -f1`. Package managers such as `yum`, `apt-get`, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see [Installing](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) and [Quick configuration with aws configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config) in the * AWS Command Line Interface User Guide*. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see [Installing AWS CLI to your home directory](https://docs.aws.amazon.com/cloudshell/latest/userguide/vm-specs.html#install-cli-software) in the * AWS CloudShell User Guide*.
+ An [IAM principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-principal) with permissions to `create` and `describe` an Amazon EKS cluster. For more information, see [Create a local Kubernetes cluster on an Outpost](security-iam-id-based-policy-examples.md#policy-create-local-cluster) and [List or describe all clusters](security-iam-id-based-policy-examples.md#policy-example2).
## Step 1: Create cluster IAM role
1. If you already have a cluster IAM role, or you’re going to create your cluster with `eksctl`, then you can skip this step. By default, `eksctl` creates a role for you.
1. Run the following command to create an IAM trust policy JSON file.
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "eks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
1. Create the Amazon EKS cluster IAM role. If necessary, preface *eks-cluster-role-trust-policy.json* with the path on your computer that you wrote the file to in the previous step. The command associates the trust policy that you created in the previous step to the role. To create an IAM role, the [IAM principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-principal) that is creating the role must be assigned the `iam:CreateRole` action (permission).
```
aws iam create-role --role-name myAmazonEKSClusterRole --assume-role-policy-document file://"eks-cluster-role-trust-policy.json"
```
1. You can assign either the Amazon EKS managed policy or create your own custom policy. For the minimum permissions that you must use in your custom policy, see [Amazon EKS cluster IAM role](cluster-iam-role.md).
Attach the Amazon EKS managed policy named [AmazonEKSClusterPolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEKSClusterPolicy.html#AmazonEKSClusterPolicy-json) to the role. To attach an IAM policy to an [IAM principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-principal), the principal that is attaching the policy must be assigned one of the following IAM actions (permissions): `iam:AttachUserPolicy` or `iam:AttachRolePolicy`.
```
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy --role-name myAmazonEKSClusterRole
```
### Service Linked Role
Amazon EKS automatically creates a service linked role called `AWSServiceRoleForAmazonEKS`.
This is in addition to the cluster IAM role. A service-linked role is a unique type of IAM role that is linked directly to Amazon EKS. The role allows Amazon EKS to manage clusters in your account. For more information, see [Using roles for Amazon EKS clusters](using-service-linked-roles-eks.md).
The IAM Identity you use to create the EKS cluster must have permission to create the service-linked role. This includes the `iam:CreateServiceLinkedRole` permission.
If the service linked role doesn’t already exist, and your current IAM role doesn’t have sufficient permissions to create it, the cluster create operation will fail.
## Step 2: Create cluster
You can create a cluster by using:
+ [`eksctl`](#step2-eksctl)
+ [the AWS Management Console](#step2-console)
+ [the AWS CLI](#step2-cli)
### Create cluster - eksctl
1. You need version `0.215.0` or later of the `eksctl` command line tool installed on your device or AWS CloudShell. To install or update `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
1. Create an Amazon EKS `IPv4` cluster with the Amazon EKS default Kubernetes version in your default AWS Region. Before running command, make the following replacements:
1. Replace *region-code* with the AWS Region that you want to create your cluster in.
1. Replace *my-cluster* with a name for your cluster. The name can contain only alphanumeric characters (case-sensitive) and hyphens. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
1. Replace *1.35* with any [Amazon EKS supported version](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
1. Change the values for `vpc-private-subnets` to meet your requirements. You can also add additional IDs. You must specify at least two subnet IDs. If you’d rather specify public subnets, you can change `--vpc-private-subnets` to `--vpc-public-subnets`. Public subnets have an associated route table with a route to an internet gateway, but private subnets don’t have an associated route table. We recommend using private subnets whenever possible.
The subnets that you choose must meet the [Amazon EKS subnet requirements](network-reqs.md#network-requirements-subnets). Before selecting subnets, we recommend that you’re familiar with all of the [Amazon EKS VPC and subnet requirements and considerations](network-reqs.md).
1. Run the following command:
```
eksctl create cluster --name my-cluster --region region-code --version 1.35 --vpc-private-subnets subnet-ExampleID1,subnet-ExampleID2 --without-nodegroup
```
Cluster provisioning takes several minutes. While the cluster is being created, several lines of output appear. The last line of output is similar to the following example line.
```
[✓] EKS cluster "my-cluster" in "region-code" region is ready
```
1. Continue with [Step 3: Update kubeconfig](#step3)
#### Optional Settings
To see the most options that you can specify when creating a cluster with `eksctl`, use the `eksctl create cluster --help` command. To see all the available options, you can use a `config` file. For more information, see [Using config files](https://eksctl.io/usage/creating-and-managing-clusters/#using-config-files) and the [config file schema](https://eksctl.io/usage/schema/) in the `eksctl` documentation. You can find [config file examples](https://github.com/weaveworks/eksctl/tree/master/examples) on GitHub.
The following are optional settings that, if required, must be added to the previous command. You can only enable these options when you create the cluster, not after. If you need to specify these options, you must create the cluster with an [eksctl config file](https://eksctl.io/usage/creating-and-managing-clusters/#using-config-files) and specify the settings, rather than using the previous command.
+ If you want to specify one or more security groups that Amazon EKS assigns to the network interfaces that it creates, specify the [securityGroup](https://eksctl.io/usage/schema/#vpc-securityGroup) option.
Whether you choose any security groups or not, Amazon EKS creates a security group that enables communication between your cluster and your VPC. Amazon EKS associates this security group, and any that you choose, to the network interfaces that it creates. For more information about the cluster security group that Amazon EKS creates, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md). You can modify the rules in the cluster security group that Amazon EKS creates.
+ If you want to specify which `IPv4` Classless Inter-domain Routing (CIDR) block Kubernetes assigns service IP addresses from, specify the [serviceIPv4CIDR](https://eksctl.io/usage/schema/#kubernetesNetworkConfig-serviceIPv4CIDR) option.
Specifying your own range can help prevent conflicts between Kubernetes services and other networks peered or connected to your VPC. Enter a range in CIDR notation. For example: `10.2.0.0/16`.
The CIDR block must meet the following requirements:
+ Be within one of the following ranges: `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`.
+ Have a minimum size of `/24` and a maximum size of `/12`.
+ Not overlap with the range of the VPC for your Amazon EKS resources.
You can only specify this option when using the `IPv4` address family and only at cluster creation. If you don’t specify this, then Kubernetes assigns service IP addresses from either the `10.100.0.0/16` or `172.20.0.0/16` CIDR blocks.
+ If you’re creating cluster and want the cluster to assign `IPv6` addresses to Pods and services instead of `IPv4` addresses, specify the [ipFamily](https://eksctl.io/usage/schema/#kubernetesNetworkConfig-ipFamily) option.
Kubernetes assigns `IPv4` addresses to Pods and services, by default. Before deciding to use the `IPv6` family, make sure that you’re familiar with all of the considerations and requirements in the [VPC requirements and considerations](network-reqs.md#network-requirements-vpc), [Subnet requirements and considerations](network-reqs.md#network-requirements-subnets), [View Amazon EKS security group requirements for clusters](sec-group-reqs.md), and [Learn about IPv6 addresses to clusters, Pods, and services](cni-ipv6.md) topics. If you choose the `IPv6` family, you can’t specify an address range for Kubernetes to assign `IPv6` service addresses from like you can for the `IPv4` family. Kubernetes assigns service addresses from the unique local address range (`fc00::/7`).
### Create cluster - AWS console
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose **Add cluster** and then choose **Create**.
1. Under **Configuration options** select **Custom configuration**
+ For information about quickly creating a cluster wih EKS Auto Mode, see [Create an EKS Auto Mode Cluster with the AWS Management Console](automode-get-started-console.md).
1. Under **EKS Auto Mode**, toggle **Use EKS Auto Mode** off.
+ For information about creating an EKS Auto Mode cluster with custom configuration, see [Create an Amazon EKS Auto Mode cluster](create-cluster-auto.md).
1. On the **Configure cluster** page, enter the following fields:
+ **Name** – A name for your cluster. The name can contain only alphanumeric characters (case-sensitive), hyphens, and underscores. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
+ **Cluster IAM role** – Choose the Amazon EKS cluster IAM role that you created to allow the Kubernetes control plane to manage AWS resources on your behalf.
+ **Kubernetes version** – The version of Kubernetes to use for your cluster. We recommend selecting the latest version, unless you need an earlier version.
+ **Support type** — The Kubernetes version policy you would like to set for your cluster. If you want your cluster to only run on a standard support version, you can choose **Standard support**. If you want your cluster to enter extended support at the end of standard support for a version, you can choose **Extended support**. If you select a Kubernetes version that is currently in extended support, you can not select standard support as an option.
+ **Secrets encryption** – (Optional) Choose to enable secrets encryption of Kubernetes secrets using a KMS key. You can also enable this after you create your cluster. Before you enable this capability, make sure that you’re familiar with the information in [Encrypt Kubernetes secrets with KMS on existing clusters](enable-kms.md).
+ **Tags** – (Optional) Add any tags to your cluster. For more information, see [Organize Amazon EKS resources with tags](eks-using-tags.md).
+ **ARC Zonal shift** - (Optional) You can use Route53 Application Recovery controller to mitigate impaired availability zones. For more information, see [Learn about Amazon Application Recovery Controller (ARC) zonal shift in Amazon EKS](zone-shift.md).
1. In the **Cluster access** section of the configure cluster page, enter the following fields:
+ **Bootstrap cluster administrator access** — The cluster creator is automatically a Kubernetes administrator. If you want to disable this, select **Disallow cluster administrator access**.
+ **Cluster authentication mode** — Determine how you want to grant IAM users and roles access to Kubernetes APIs. For more information, see [Set Cluster Authentication Mode](grant-k8s-access.md#set-cam).
When you’re done with this page, choose **Next**.
1. On the **Specify networking** page, select values for the following fields:
+ **VPC** – Choose an existing VPC that meets [Amazon EKS VPC requirements](network-reqs.md#network-requirements-vpc) to create your cluster in. Before choosing a VPC, we recommend that you’re familiar with all of the requirements and considerations in [View Amazon EKS networking requirements for VPC and subnets](network-reqs.md). You can’t change which VPC you want to use after cluster creation. If no VPCs are listed, then you need to create one first. For more information, see [Create an Amazon VPC for your Amazon EKS cluster](creating-a-vpc.md).
+ **Subnets** – By default, all available subnets in the VPC specified in the previous field are preselected. You must select at least two.
The subnets that you choose must meet the [Amazon EKS subnet requirements](network-reqs.md#network-requirements-subnets). Before selecting subnets, we recommend that you’re familiar with all of the [Amazon EKS VPC and subnet requirements and considerations](network-reqs.md).
**Security groups** – (Optional) Specify one or more security groups that you want Amazon EKS to associate to the network interfaces that it creates.
Whether you choose any security groups or not, Amazon EKS creates a security group that enables communication between your cluster and your VPC. Amazon EKS associates this security group, and any that you choose, to the network interfaces that it creates. For more information about the cluster security group that Amazon EKS creates, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md). You can modify the rules in the cluster security group that Amazon EKS creates.
+ **Choose cluster IP address family** – You can choose either **IPv4** and **IPv6**.
Kubernetes assigns `IPv4` addresses to Pods and services, by default. Before deciding to use the `IPv6` family, make sure that you’re familiar with all of the considerations and requirements in the [VPC requirements and considerations](network-reqs.md#network-requirements-vpc), [Subnet requirements and considerations](network-reqs.md#network-requirements-subnets), [View Amazon EKS security group requirements for clusters](sec-group-reqs.md), and [Learn about IPv6 addresses to clusters, Pods, and services](cni-ipv6.md) topics. If you choose the `IPv6` family, you can’t specify an address range for Kubernetes to assign `IPv6` service addresses from like you can for the `IPv4` family. Kubernetes assigns service addresses from the unique local address range (`fc00::/7`).
+ (Optional) Choose **Configure Kubernetes Service IP address range** and specify a **Service `IPv4` range**.
Specifying your own range can help prevent conflicts between Kubernetes services and other networks peered or connected to your VPC. Enter a range in CIDR notation. For example: `10.2.0.0/16`.
The CIDR block must meet the following requirements:
+ Be within one of the following ranges: `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`.
+ Have a minimum size of `/24` and a maximum size of `/12`.
+ Not overlap with the range of the VPC for your Amazon EKS resources.
You can only specify this option when using the `IPv4` address family and only at cluster creation. If you don’t specify this, then Kubernetes assigns service IP addresses from either the `10.100.0.0/16` or `172.20.0.0/16` CIDR blocks.
+ For **Cluster endpoint access**, select an option. After your cluster is created, you can change this option. Before selecting a non-default option, make sure to familiarize yourself with the options and their implications. For more information, see [Cluster API server endpoint](cluster-endpoint.md).
When you’re done with this page, choose **Next**.
1. (Optional) On the **Configure observability** page, choose which **Metrics** and **Control plane logging** options to turn on. By default, each log type is turned off.
+ For more information about the Prometheus metrics option, see [Step 1: Turn on Prometheus metrics](prometheus.md#turn-on-prometheus-metrics).
+ For more information about the **Control plane logging** options, see [Send control plane logs to CloudWatch Logs](control-plane-logs.md).
When you’re done with this page, choose **Next**.
1. On the **Select add-ons** page, choose the add-ons that you want to add to your cluster. Certain add-ons are pre-selected. You can choose as many **Amazon EKS add-ons** and ** AWS Marketplace add-ons** as you require. If the ** AWS Marketplace add-ons** that you want to install isn’t listed, you can click the page numbering to view additional page results or search for available ** AWS Marketplace add-ons** by entering text in the search box. You can also filter by **category**, **vendor**, or **pricing model** and then choose the add-ons from the search results. When creating a cluster, you can view, select, and install any add-on that supports EKS Pod Identities as detailed in [Learn how EKS Pod Identity grants pods access to AWS services](pod-identities.md).
When you’re done with this page, choose **Next**.
Some add-ons, such as Amazon VPC CNI, CoreDNS, and kube-proxy, are installed by default. If you disable any of the default add-ons, this may affect your ability to run Kubernetes applications.
1. On the **Configure selected add-ons settings** page, select the version that you want to install. You can always update to a later version after cluster creation.
For add-ons that support EKS Pod Identities, you can use the console to automatically generate the role with the name, AWS managed policy, and trust policy prepopulated specifically for the add-on. You can re-use existing roles or create new roles for supported add-ons. For the steps to use the console to create roles for add-ons that support EKS Pod Identities, see [Create add-on (AWS Console)](creating-an-add-on.md#create_add_on_console). If an add-on does not support EKS Pod Identity, a message displays with instructions to use the wizard to create the IAM roles for service accounts (IRSA) after the cluster is created.
You can update the configuration of each add-on after cluster creation. For more information about configuring add-ons, see [Update an Amazon EKS add-on](updating-an-add-on.md). When you’re done with this page, choose **Next**.
1. On the **Review and create** page, review the information that you entered or selected on the previous pages. If you need to make changes, choose **Edit**. When you’re satisfied, choose **Create**. The **Status** field shows **CREATING** while the cluster is provisioned.
**Note**
You might receive an error that one of the Availability Zones in your request doesn’t have sufficient capacity to create an Amazon EKS cluster. If this happens, the error output contains the Availability Zones that can support a new cluster. Retry creating your cluster with at least two subnets that are located in the supported Availability Zones for your account. For more information, see [Insufficient capacity](troubleshooting.md#ice).
Cluster provisioning takes several minutes.
1. Continue with [Step 3: Update kubeconfig](#step3)
### Create cluster - AWS CLI
1. Create your cluster with the command that follows. Before running the command, make the following replacements:
+ Replace *region-code* with the AWS Region that you want to create your cluster in.
+ Replace *my-cluster* with a name for your cluster. The name can contain only alphanumeric characters (case-sensitive), hyphens, and underscores. It must start with an alphanumeric character and can’t be longer than 100 characters. The name must be unique within the AWS Region and AWS account that you’re creating the cluster in.
+ Replace *1.35* with any [Amazon EKS supported version](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
+ Replace *111122223333* with your account ID and *myAmazonEKSClusterRole* with the name of your cluster IAM role.
+ Replace the values for `subnetIds` with your own. You can also add additional IDs. You must specify at least two subnet IDs.
The subnets that you choose must meet the [Amazon EKS subnet requirements](network-reqs.md#network-requirements-subnets). Before selecting subnets, we recommend that you’re familiar with all of the [Amazon EKS VPC and subnet requirements and considerations](network-reqs.md).
+ If you don’t want to specify a security group ID, remove `,securityGroupIds=sg-` from the command. If you want to specify one or more security group IDs, replace the values for `securityGroupIds` with your own. You can also add additional IDs.
Whether you choose any security groups or not, Amazon EKS creates a security group that enables communication between your cluster and your VPC. Amazon EKS associates this security group, and any that you choose, to the network interfaces that it creates. For more information about the cluster security group that Amazon EKS creates, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md). You can modify the rules in the cluster security group that Amazon EKS creates.
```
aws eks create-cluster --region region-code --name my-cluster --kubernetes-version 1.35 \
--role-arn arn:aws:iam::111122223333:role/myAmazonEKSClusterRole \
--resources-vpc-config subnetIds=subnet-ExampleID1,subnet-ExampleID2,securityGroupIds=sg-ExampleID1
```
**Note**
You might receive an error that one of the Availability Zones in your request doesn’t have sufficient capacity to create an Amazon EKS cluster. If this happens, the error output contains the Availability Zones that can support a new cluster. Retry creating your cluster with at least two subnets that are located in the supported Availability Zones for your account. For more information, see [Insufficient capacity](troubleshooting.md#ice).
The following are optional settings that, if required, must be added to the previous command. You can only enable these options when you create the cluster, not after.
+ By default, EKS installs multiple networking add-ons during cluster creation. This includes the Amazon VPC CNI, CoreDNS, and kube-proxy.
If you’d like to disable the installation of these default networking add-ons, use the parameter below. This may be used for alternate CNIs, such as Cilium. Review the [EKS API reference](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateCluster.html) for more information.
`aws eks create-cluster --no-bootstrap-self-managed-addons …`
+ If you want to specify which `IPv4` Classless Inter-domain Routing (CIDR) block Kubernetes assigns service IP addresses from, you must specify it by adding the `--kubernetes-network-config serviceIpv4Cidr=` to the following command.
Specifying your own range can help prevent conflicts between Kubernetes services and other networks peered or connected to your VPC. Enter a range in CIDR notation. For example: `10.2.0.0/16`.
The CIDR block must meet the following requirements:
+ Be within one of the following ranges: `10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`.
+ Have a minimum size of `/24` and a maximum size of `/12`.
+ Not overlap with the range of the VPC for your Amazon EKS resources.
You can only specify this option when using the `IPv4` address family and only at cluster creation. If you don’t specify this, then Kubernetes assigns service IP addresses from either the `10.100.0.0/16` or `172.20.0.0/16` CIDR blocks.
+ If you’re creating a cluster and want the cluster to assign `IPv6` addresses to Pods and services instead of `IPv4` addresses, add `--kubernetes-network-config ipFamily=ipv6` to the following command.
Kubernetes assigns `IPv4` addresses to Pods and services, by default. Before deciding to use the `IPv6` family, make sure that you’re familiar with all of the considerations and requirements in the [VPC requirements and considerations](network-reqs.md#network-requirements-vpc), [Subnet requirements and considerations](network-reqs.md#network-requirements-subnets), [View Amazon EKS security group requirements for clusters](sec-group-reqs.md), and [Learn about IPv6 addresses to clusters, Pods, and services](cni-ipv6.md) topics. If you choose the `IPv6` family, you can’t specify an address range for Kubernetes to assign `IPv6` service addresses from like you can for the `IPv4` family. Kubernetes assigns service addresses from the unique local address range (`fc00::/7`).
1. It takes several minutes to provision the cluster. You can query the status of your cluster with the following command.
```
aws eks describe-cluster --region region-code --name my-cluster --query "cluster.status"
```
Don’t proceed to the next step until the output returned is `ACTIVE`.
1. Continue with [Step 3: Update kubeconfig](#step3)
## Step 3: Update kubeconfig
1. If you created your cluster using `eksctl`, then you can skip this step. This is because `eksctl` already completed this step for you. Enable `kubectl` to communicate with your cluster by adding a new context to the `kubectl` `config` file. For more information about how to create and update the file, see [Connect kubectl to an EKS cluster by creating a kubeconfig file](create-kubeconfig.md).
```
aws eks update-kubeconfig --region region-code --name my-cluster
```
An example output is as follows.
```
Added new context arn:aws:eks:region-code:111122223333:cluster/my-cluster to /home/username/.kube/config
```
1. Confirm communication with your cluster by running the following command.
```
kubectl get svc
```
An example output is as follows.
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 443/TCP 28h
```
## Step 4: Cluster setup
1. (Recommended) To use some Amazon EKS add-ons, or to enable individual Kubernetes workloads to have specific AWS Identity and Access Management (IAM) permissions, [create an IAM OpenID Connect (OIDC) provider](enable-iam-roles-for-service-accounts.md) for your cluster. You only need to create an IAM OIDC provider for your cluster once. To learn more about Amazon EKS add-ons, see [Amazon EKS add-ons](eks-add-ons.md). To learn more about assigning specific IAM permissions to your workloads, see [IAM roles for service accounts](iam-roles-for-service-accounts.md).
1. (Recommended) Configure your cluster for the Amazon VPC CNI plugin for Kubernetes plugin before deploying Amazon EC2 nodes to your cluster. By default, the plugin was installed with your cluster. When you add Amazon EC2 nodes to your cluster, the plugin is automatically deployed to each Amazon EC2 node that you add. The plugin requires you to attach one of the following IAM policies to an IAM role. If your cluster uses the `IPv4` family, use the [AmazonEKS\$1CNI\$1Policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEKS_CNI_Policy.html) managed IAM policy. If your cluster uses the `IPv6` family, use an [IAM policy that you create](cni-iam-role.md#cni-iam-role-create-ipv6-policy).
The IAM role that you attach the policy to can be the node IAM role, or a dedicated role used only for the plugin. We recommend attaching the policy to this role. For more information about creating the role, see [Configure Amazon VPC CNI plugin to use IRSA](cni-iam-role.md) or [Amazon EKS node IAM role](create-node-role.md).
1. If you deployed your cluster using the AWS Management Console, you can skip this step. The AWS Management Console deploys the Amazon VPC CNI plugin for Kubernetes, CoreDNS, and `kube-proxy` Amazon EKS add-ons, by default.
If you deploy your cluster using either `eksctl` or the AWS CLI, then the Amazon VPC CNI plugin for Kubernetes, CoreDNS, and `kube-proxy` self-managed add-ons are deployed. You can migrate the Amazon VPC CNI plugin for Kubernetes, CoreDNS, and `kube-proxy` self-managed add-ons that are deployed with your cluster to Amazon EKS add-ons. For more information, see [Amazon EKS add-ons](eks-add-ons.md).
1. (Optional) If you haven’t already done so, you can enable Prometheus metrics for your cluster. For more information, see [Create a scraper](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-collector-how-to.html#AMP-collector-create) in the *Amazon Managed Service for Prometheus User Guide*.
1. If you plan to deploy workloads to your cluster that use Amazon EBS volumes, then you must install the [Amazon EBS CSI](ebs-csi.md) to your cluster before deploying the workloads.
## Next steps
+ The [IAM principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-principal) that created the cluster is the only principal that has access to the cluster. [Grant permissions to other IAM principals](grant-k8s-access.md) so they can access your cluster.
+ If the IAM principal that created the cluster only has the minimum IAM permissions referenced in the prerequisites, then you might want to add additional Amazon EKS permissions for that principal. For more information about granting Amazon EKS permissions to IAM principals, see [Identity and access management for Amazon EKS](security-iam.md).
+ If you want the IAM principal that created the cluster, or any other principals to view Kubernetes resources in the Amazon EKS console, grant the [Required permissions](view-kubernetes-resources.md#view-kubernetes-resources-permissions) to the entities.
+ If you want nodes and IAM principals to access your cluster from within your VPC, enable the private endpoint for your cluster. The public endpoint is enabled by default. You can disable the public endpoint once you’ve enabled the private endpoint, if desired. For more information, see [Cluster API server endpoint](cluster-endpoint.md).
+ [Enable secrets encryption for your cluster](enable-kms.md).
+ [Configure logging for your cluster](control-plane-logs.md).
+ [Add nodes to your cluster](eks-compute.md).
# Amazon EKS Provisioned Control Plane
## Overview
Amazon EKS Provisioned Control Plane is a feature that enables cluster administrators to select from a set of scaling tiers and designate their chosen tier for very high, predictable performance from the cluster’s control plane. This enables cluster administrators to ensure that the control plane is always provisioned with the specified capacity.
Amazon EKS offers two modes of operations for your cluster’s control plane. By default, Amazon EKS clusters use **Standard mode**, where the control plane automatically scales up and down based on your workload demands. Standard mode dynamically allocates sufficient control plane capacity to meet your workload needs and is the recommended solution for most use cases. However, for specialized workloads that cannot tolerate any performance variability due to control plane scaling or those requiring very high amounts of control plane capacity, you can optionally use **Provisioned mode**. Provisioned mode allows you to pre-allocate control plane capacity that is always ready to handle demanding workload requirements.
**Note**
Provisioned mode is an additional control plane operations mode alongside the default Standard mode. The introduction of Provisioned mode does not change Standard mode behavior.
With EKS Provisioned Control Plane, cluster administrators can pre-provision the desired control plane capacity ahead of time, providing predictable and high performance from the cluster’s control plane that is always available. EKS Provisioned Control Plane also enables cluster administrators to provision the same control plane capacity across environments, from staging to production and disaster recovery sites. This is important for ensuring that the control plane performance obtained across environments is consistent and predictable. Finally, EKS Provisioned Control Plane gives you access to very high levels of control plane performance, enabling the running of massively scalable AI workloads, high-performance computing, and large-scale data processing workloads on Kubernetes.
All existing and new Amazon EKS clusters operate in Standard mode by default. For clusters requiring high, predictable performance from the control plane, you can opt in to use the EKS Provisioned Control Plane feature. You will be billed at the hourly rate for the particular control plane scaling tier in addition to the standard or extended support EKS hourly charges. For more information about pricing, see [Amazon EKS pricing](https://aws.amazon.com/eks/pricing/).
![\[Amazon EKS Control Plane Modes\]](http://docs.aws.amazon.com/eks/latest/userguide/images/control-plane-modes.png)
## Use cases
EKS Provisioned Control Plane is designed to address specific scenarios where high and predictable control plane performance is critical to your operations. Understanding these use cases can help you determine whether EKS Provisioned Control Plane is the right solution for your workloads.
**Performance-critical workloads** – For workloads that demand minimal latency and maximum performance from the Kubernetes control plane, EKS Provisioned Control Plane provides capacity that eliminates performance variability with control plane scaling.
**Massively scalable workloads** – If you run highly scalable workloads such as AI training and inference, high-performance computing, or large-scale data processing that require a large number of nodes running in the cluster, Provisioned Control Plane provides the necessary control plane capacity to support these demanding workloads.
**Anticipated high-demand events** – When you expect a sudden surge in control plane requests due to an upcoming event such as e-commerce sales or promotions, product launches, holiday shopping seasons, or major sporting or entertainment events, Provisioned Control Plane allows you to scale your control plane capacity in advance. This proactive approach ensures your control plane is ready to handle the increased load without waiting for automatic scaling to respond to demand.
**Mission-critical high availability** - For workloads that demand the highest levels of availability, Provisioned Control Plane offers a [99.99% Service Level Agreement (SLA)](https://aws.amazon.com/eks/sla/), measured in 1-minute intervals. This enhanced SLA provides a more stringent availability commitment compared to the 99.95% SLA of Standard control plane mode, measured in 5-minute intervals.
**Environment consistency** – Provisioned Control Plane enables you to match control plane capacity and performance across staging and production environments, helping you identify potential issues early before deployment to production. By maintaining the same control plane tier across environments, you can ensure that testing results accurately reflect production behavior, reducing the risk of performance-related surprises during rollout.
**Disaster recovery and business continuity** – For disaster recovery scenarios, Provisioned Control Plane allows you to provision failover environments with the same level of capacity as your primary environment. This ensures minimal disruption and quick recovery during failover events, as your disaster recovery cluster will have identical control plane performance characteristics to your production cluster from the moment it’s activated.
## Control Plane Scaling Tiers
EKS Provisioned Control Plane offers scaling tiers named using t-shirt sizes (XL, 2XL, 4XL, and 8XL). Each tier defines its capability through four key attributes that determine the performance characteristics of your cluster’s control plane. Understanding these attributes helps you select the appropriate tier for your workload requirements.
**API request concurrency** measures the number of requests that the Kubernetes control plane’s API server can process concurrently, which is critical for high throughput workloads.
**Pod scheduling rate** indicates how quickly the default Kubernetes scheduler can schedule pods on nodes, measured in pods per second.
**Cluster database size** indicates the storage space allocated to etcd, the database that holds the cluster state/metadata.
**SLA (Service Level Agreement)** indicates the monthly uptime commitment of the Kubernetes endpoint of the EKS cluster
When you provision your cluster’s control plane on a certain scaling tier using Provisioned Control Plane, EKS ensures your cluster’s control plane maintains the limits corresponding to that tier. The limits of control plane scaling tiers vary by Kubernetes version, as shown in the following tables.
### EKS v1.30 to v1.33
| Provisioned Control Plane Scaling Tier | API request concurrency (seats) | Pod scheduling rate (pods/sec) | Cluster database size (GB) | SLA (measured in 1-minute intervals) |
| --- | --- | --- | --- | --- |
| XL | 1700 | 167 | 16 | 99.99% |
| 2XL | 3400 | 283 | 16 | 99.99% |
| 4XL | 6800 | 400 | 16 | 99.99% |
| 8XL | 13600 | 400 | 16 | 99.99% |
### EKS v1.34 and later
| Provisioned Control Plane Scaling Tier | API request concurrency (seats) | Pod scheduling rate (pods/sec) | Cluster database size (GB) | SLA (measured in 1-minute intervals) |
| --- | --- | --- | --- | --- |
| XL | 2000 | 167 | 16 | 99.99% |
| 2XL | 4000 | 283 | 16 | 99.99% |
| 4XL | 8000 | 400 | 16 | 99.99% |
| 8XL | 16000 | 400 | 16 | 99.99% |
### Monitoring control plane scaling tier utilization
Amazon EKS provides several metrics to help you monitor your control plane’s tier utilization. These metrics are published as [Amazon CloudWatch metrics](cloudwatch.md) and are accessible through the CloudWatch and EKS console. Additionally, these metrics are scrapable from your EKS cluster’s Prometheus endpoint (see [here](prometheus.md)).
| | **Prometheus Metric** | **CloudWatch Metric** |
| --- | --- | --- |
| **API request concurrency** | apiserver\$1flowcontrol\$1current\$1executing\$1seats | apiserver\$1flowcontrol\$1current\$1executing\$1seats |
| **Pod scheduling rate** | scheduler\$1schedule\$1attempts\$1total | scheduler\$1schedule\$1attempts\$1total, scheduler\$1schedule\$1attempts\$1SCHEDULED, scheduler\$1schedule\$1attempts\$1UNSCHEDULABLE |
| **Cluster database size** | apiserver\$1storage\$1size\$1bytes (NOTE: etcd\$1mvcc\$1db\$1total\$1size\$1in\$1use\$1in\$1bytes will be rolled out as a prometheus metric to all EKS clusters by second half of 2026) | etcd\$1mvcc\$1db\$1total\$1size\$1in\$1use\$1in\$1bytes |
You can view control plane utilization in the Amazon EKS console. From your cluster’s overview page, choose **Monitor cluster** to access the observability dashboard, then select the **Control plane monitoring** tab to view control plane utilization under the **Control plane scaling** section.
![\[Monitor EKS cluster\]](http://docs.aws.amazon.com/eks/latest/userguide/images/monitor-cluster.png)
![\[EKS Control Plane Monitoring\]](http://docs.aws.amazon.com/eks/latest/userguide/images/control-plane-monitoring.png)
### Understanding Tier capacity versus actual performance
When you select a Provisioned Control Plane scaling tier, the tier attributes represent the underlying configurations that Amazon EKS applies to your control plane. However, the actual performance you achieve depends on your specific workload patterns, configurations, and adherence to Kubernetes best practices. For example, while a 4XL tier configures API Priority and Fairness (APF) with 6,800 concurrent request seats, the actual request throughput you obtain from the control plane depends on the types of operations being performed. For example, Kubernetes penalizes list requests more than get, and hence the effective number of list requests processed concurrently by control plane is lower than get requests (see [here](https://docs.aws.amazon.com/eks/latest/best-practices/scale-control-plane.html#_api_priority_and_fairness)). Similarly, although the default scheduler QPS is set to 400 for a 4XL tier, your actual pod scheduling rate depends on factors like nodes being ready and healthy for scheduling. To achieve optimal performance, ensure your applications follow Kubernetes best practices (see [here](https://docs.aws.amazon.com/eks/latest/best-practices/scalability.html)) and are properly configured for your workload characteristics.
## Considerations
+ **Standard control plane capacity** – EKS Standard control plane mode offers the best price to performance ratio, and is the recommended option for the vast majority of use cases. However, for specialized workloads that cannot tolerate any performance variability due to control plane scaling or those requiring very high amounts of control plane capacity, you can optionally consider using Provisioned mode.
+ **Opt-in required** – Existing clusters will not automatically scale up from the Standard control plane to a higher [priced](https://aws.amazon.com/eks/pricing/) EKS Provisioned Control Plane tier. You must explicitly opt in to one of the new EKS Provisioned Control Plane scaling tiers.
+ **Exit restriction** – Standard control plane mode supports up to 8 GB of cluster database (etcd) size. If your cluster’s database size exceeds 8 GB while using Provisioned mode, you cannot switch back to Standard mode until you reduce the database size to below 8 GB. For example, if you are using 14 GB of database storage in Provisioned mode, you must first reduce your database utilization to less than 8GB before returning to Standard mode.
+ **No automatic tier scaling** – EKS Provisioned Control Plane does not automatically scale between tiers. Once you select a scaling tier, your cluster’s control plane remains pinned to that tier, ensuring consistent and predictable performance. However, you have the flexibility to implement your own autoscaling solution by monitoring tier utilization metrics and using the EKS Provisioned Control Plane APIs to scale up or down when these metrics cross thresholds you define, giving you full control over your scaling strategy and cost optimization.
+ **Viewing current tier** – You can use the Amazon EKS console, Amazon Web Services CLI, or API to view the current control plane scaling tier. In the CLI, you can run the `describe-cluster` command: `aws eks describe-cluster --name cluster-name`
+ **Tier transition time** – You can use the Amazon EKS console, Amazon EKS APIs, or CLI to exit or move between scaling tiers. Amazon EKS has introduced a new cluster update type called `ScalingTierConfigUpdate`, which you can inspect to monitor the progress of the transition. After you execute a tier change command, you can list the updates on the cluster to see a new update of type `ScalingTierConfigUpdate` with status `Updating`. The status changes to `Successful` upon completion of the update, or to `Failed` if an error occurs. The error field in the update indicates the reason for failure. There are no restrictions on how frequently you can switch between tiers. Changing the control plane tier takes several minutes to complete. There is no API server downtime during this process, as EKS brings up new API servers before terminating old ones.
+ **Selecting optimal tier** – To determine the optimal Provisioned Control Plane scaling tier for your cluster, you can perform load testing by provisioning your cluster on the highest tier (8XL). Then perform a load test to simulate peak demand on your cluster’s control plane. Observe the control plane tier utilization metrics at peak load, and use these observations as the guiding factor to select the appropriate tier for Provisioned mode.
+ **Provisioned Control Plane pricing** – You will be billed at the hourly rate for the Provisioned Control Plane scaling tier your cluster is on. This is in addition to the standard or extended support hourly charges. See Amazon EKS Pricing [page](https://aws.amazon.com/eks/pricing/) for details.
+ **Larger scaling tier** – If you intend to run your cluster on scaling tier larger than 8XL, contact your Amazon Web Services account team for additional pricing information.
+ **Kubernetes version and region support** – EKS Provisioned Control Plane is supported in all Amazon Web Services commercial, GovCloud, and China regions. Provisioned Control Plane works on EKS v1.28 and higher.
+ **Higher SLA** - EKS Provisioned Control Plane offers higher EKS SLA than Standard Control Plane. Learn more in [EKS SLA page](https://aws.amazon.com/eks/sla/).
# Getting started with the Amazon EKS Provisioned Control Plane
This guide walks you through the steps to set up and use EKS Provisioned Control Plane using AWS CLI and AWS Management Console.
## Prerequisites
Before you begin, ensure you have:
+ ** AWS CLI** – A command line tool for working with AWS services, including Amazon EKS. For more information, see [Installing](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) in the * AWS Command Line Interface User Guide*. After installing the AWS CLI, we recommend that you also configure it. For more information, see [Quick configuration with aws configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config) in the * AWS Command Line Interface User Guide*. Note that AWS CLI v2 is required to use the **update-kubeconfig** option shown in this page.
+ **Required IAM permissions** – The IAM security principal that you’re using must have permissions to work with Amazon EKS IAM roles, service linked roles, AWS CloudFormation, a VPC, and related resources. For more information, see [Actions](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonelastickubernetesservice.html) and [Using service-linked roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html) in the *IAM User Guide*. You must complete all steps in this guide as the same user. To check the current user, run the following command:
```
aws sts get-caller-identity
```
**Note**
We recommend that you complete the steps in this topic in a Bash shell. If you aren’t using a Bash shell, some script commands such as line continuation characters and the way variables are set and used require adjustment for your shell. Additionally, the quoting and escaping rules for your shell might be different. For more information, see [Using quotation marks with strings in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-parameters-quoting-strings.html) in the * AWS Command Line Interface User Guide*.
## EKS Provisioned Control Plane - AWS CLI
### Create cluster with EKS Provisioned Control Plane Scaling Tier
```
aws eks create-cluster --name prod-cluster \
--role-arn arn:aws:iam::012345678910:role/eks-service-role-AWSServiceRoleForAmazonEKS-J7ONKE3BQ4PI \
--resources-vpc-config subnetIds=subnet-6782e71e,subnet-e7e761ac,securityGroupIds=sg-6979fe18 \
--control-plane-scaling-config tier=tier-xl
```
Response:
```
{
"cluster": {
"name": "my-eks-cluster",
"arn": "arn:aws:eks:us-east-2:111122223333:cluster/my-eks-cluster",
"createdAt": "2024-03-14T11:31:44.348000-04:00",
"version": "1.26",
"endpoint": "https://JSA79429HJDASKJDJ8223829MNDNASW.yl4.us-east-2.eks.amazonaws.com",
"roleArn": "arn:aws:iam::111122223333:role/eksctl-my-eks-cluster-cluster-ServiceRole-zMF6CBakwwbW",
"resourcesVpcConfig": {
"subnetIds": [
"subnet-0fb75d2d8401716e7",
"subnet-02184492f67a3d0f9",
"subnet-04098063527aab776",
"subnet-0e2907431c9988b72",
"subnet-04ad87f71c6e5ab4d",
"subnet-09d912bb63ef21b9a"
],
"securityGroupIds": [
"sg-0c1327f6270afbb36"
],
"clusterSecurityGroupId": "sg-01c84d09d70f39a7f",
"vpcId": "vpc-0012b8e1cc0abb17d",
"endpointPublicAccess": true,
"endpointPrivateAccess": true,
"publicAccessCidrs": [
"22.19.18.2/32"
]
},
"controlPlaneScalingConfig": {
"tier": "tier-xl"
},
"kubernetesNetworkConfig": {
"serviceIpv4Cidr": "10.100.0.0/16",
"ipFamily": "ipv4"
},
"logging": {
"clusterLogging": [
{
"types": [
"api",
"audit",
"authenticator",
"controllerManager",
"scheduler"
],
"enabled": true
}
]
},
"identity": {
"oidc": {
"issuer": "https://oidc.eks.us-east-2.amazonaws.com/id/JSA79429HJDASKJDJ8223829MNDNASW"
}
},
"status": "CREATING",
"certificateAuthority": {
"data": "CA_DATA_STRING..."
},
"platformVersion": "eks.14",
"tags": {
"aws:cloudformation:stack-name": "eksctl-my-eks-cluster-cluster",
"alpha.eksctl.io/cluster-name": "my-eks-cluster",
"karpenter.sh/discovery": "my-eks-cluster",
"aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-2:111122223333:stack/eksctl-my-eks-cluster-cluster/e752ea00-e217-11ee-beae-0a9599c8c7ed",
"auto-delete": "no",
"eksctl.cluster.k8s.io/v1alpha1/cluster-name": "my-eks-cluster",
"EKS-Cluster-Name": "my-eks-cluster",
"alpha.eksctl.io/cluster-oidc-enabled": "true",
"aws:cloudformation:logical-id": "ControlPlane",
"alpha.eksctl.io/eksctl-version": "0.173.0-dev+a7ee89342.2024-03-01T03:40:57Z",
"Name": "eksctl-my-eks-cluster-cluster/ControlPlane"
},
"health": {
"issues": []
},
"accessConfig": {
"authenticationMode": "API_AND_CONFIG_MAP"
}
}
}
```
### View cluster’s Control Plane Scaling Tier
```
aws eks describe-cluster --name prod-cluster
```
Response:
```
{
"cluster": {
"name": "my-eks-cluster",
"arn": "arn:aws:eks:us-east-2:111122223333:cluster/my-eks-cluster",
"createdAt": "2024-03-14T11:31:44.348000-04:00",
"version": "1.26",
"endpoint": "https://JSA79429HJDASKJDJ8223829MNDNASW.yl4.us-east-2.eks.amazonaws.com",
"roleArn": "arn:aws:iam::111122223333:role/eksctl-my-eks-cluster-cluster-ServiceRole-zMF6CBakwwbW",
"resourcesVpcConfig": {
"subnetIds": [
"subnet-0fb75d2d8401716e7",
"subnet-02184492f67a3d0f9",
"subnet-04098063527aab776",
"subnet-0e2907431c9988b72",
"subnet-04ad87f71c6e5ab4d",
"subnet-09d912bb63ef21b9a"
],
"securityGroupIds": [
"sg-0c1327f6270afbb36"
],
"clusterSecurityGroupId": "sg-01c84d09d70f39a7f",
"vpcId": "vpc-0012b8e1cc0abb17d",
"endpointPublicAccess": true,
"endpointPrivateAccess": true,
"publicAccessCidrs": [
"22.19.18.2/32"
]
},
"controlPlaneScalingConfig": {
"tier": "tier-xl"
},
"kubernetesNetworkConfig": {
"serviceIpv4Cidr": "10.100.0.0/16",
"ipFamily": "ipv4"
},
"logging": {
"clusterLogging": [
{
"types": [
"api",
"audit",
"authenticator",
"controllerManager",
"scheduler"
],
"enabled": true
}
]
},
"identity": {
"oidc": {
"issuer": "https://oidc.eks.us-east-2.amazonaws.com/id/JSA79429HJDASKJDJ8223829MNDNASW"
}
},
"status": "ACTIVE",
"certificateAuthority": {
"data": "CA_DATA_STRING..."
},
"platformVersion": "eks.14",
"tags": {
"aws:cloudformation:stack-name": "eksctl-my-eks-cluster-cluster",
"alpha.eksctl.io/cluster-name": "my-eks-cluster",
"karpenter.sh/discovery": "my-eks-cluster",
"aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-2:111122223333:stack/eksctl-my-eks-cluster-cluster/e752ea00-e217-11ee-beae-0a9599c8c7ed",
"auto-delete": "no",
"eksctl.cluster.k8s.io/v1alpha1/cluster-name": "my-eks-cluster",
"EKS-Cluster-Name": "my-eks-cluster",
"alpha.eksctl.io/cluster-oidc-enabled": "true",
"aws:cloudformation:logical-id": "ControlPlane",
"alpha.eksctl.io/eksctl-version": "0.173.0-dev+a7ee89342.2024-03-01T03:40:57Z",
"Name": "eksctl-my-eks-cluster-cluster/ControlPlane"
},
"health": {
"issues": []
},
"accessConfig": {
"authenticationMode": "API_AND_CONFIG_MAP"
}
}
}
```
### Update cluster to use EKS Provisioned Control Plane
```
aws eks update-cluster-config --name prod-cluster \
--control-plane-scaling-config tier=tier-2xl
```
Response:
```
{
"update": {
"id": "7551c64b-1d27-4b1e-9f8e-c45f056eb6fd",
"status": "InProgress",
"type": "ScalingTierConfigUpdate",
"params": [
{
"type": "UpdatedTier",
"value": "tier-2xl"
},
{
"type": "PreviousTier",
"value": "tier-xl"
}
],
"createdAt": 1565807210.37,
"errors": []
}
}
```
### View Control Plane Scaling update
```
aws eks list-updates --name example
```
Response:
```
{
"updateIds": [
"7551c64b-1d27-4b1e-9f8e-c45f056eb6fd1"
]
}
```
### Exit Provisioned Control Plane to Standard Control Plane
```
aws eks update-cluster-config --name prod-cluster \
--control-plane-scaling-config tier=standard
```
Response:
```
{
"update": {
"id": "7551c64b-1d27-4b1e-9f8e-c45f056eb6fd",
"status": "InProgress",
"type": "ScalingTierConfigUpdate",
"params": [
{
"type": "UpdatedTier",
"value": "standard"
},
{
"type": "PreviousTier",
"value": "tier-2xl"
}
],
"createdAt": 1565807210.37,
"errors": []
}
}
```
## EKS Provisioned Control Plane - AWS Management Console
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose **Create cluster**.
1. Under *Configuration options*, select **Custom configuration**.
1. Scroll down to **Control plane scaling tier**. Select **Use a scaling tier** to enable Provisioned Control Plane.
1. Select the control plane scaling tier that you would like to provision for the cluster from various scaling tier options such as XL, 2XL, 4XL, and 8XL.
1. Select other cluster configurations options as needed. On the final step select **Create cluster**. Note it may take several minutes for cluster creation to complete.
# Prepare for Kubernetes version upgrades and troubleshoot misconfigurations with cluster insights
Amazon EKS cluster insights provide detection of issues and recommendations to resolve them to help you manage your cluster. Every Amazon EKS cluster undergoes automatic, recurring checks against an Amazon EKS curated list of insights. These *insight checks* are fully managed by Amazon EKS and offer recommendations on how to address any findings.
## Cluster insight types
+ **Configuration insights**: Identifies misconfigurations in your EKS Hybrid Nodes setup that could impair functionality of your cluster or workloads.
+ **Upgrade insights**: Identifies issues that could impact your ability to upgrade to new versions of Kubernetes.
## Considerations
+ **Frequency**: Amazon EKS refreshes cluster insights every 24 hours, or you can manually refresh them to see the latest status. For example, you can manually refresh cluster insights after addressing an issue to see if the issue was resolved.
+ **Permissions**: Amazon EKS automatically creates a cluster access entry for cluster insights in every EKS cluster. This entry gives EKS permission to view information about your cluster. Amazon EKS uses this information to generate the insights. For more information, see [AmazonEKSClusterInsightsPolicy](access-policy-permissions.md#access-policy-permissions-AmazonEKSClusterInsightsPolicy).
## Use cases
Cluster insights in Amazon EKS provide automated checks to help maintain the health, reliability, and optimal configuration of your Kubernetes clusters. Below are key use cases for cluster insights, including upgrade readiness and configuration troubleshooting.
### Upgrade insights
Upgrade insights are a specific type of insight checks within cluster insights. These checks returns insights related to Kubernetes version upgrade readiness. Amazon EKS runs upgrade insight checks on every EKS cluster.
**Important**
Amazon EKS has temporarily rolled back a feature that would require you to use a `--force` flag to upgrade your cluster when there were certain cluster insight issues. For more information, see [Temporary rollback of enforcing upgrade insights on update cluster version](https://github.com/aws/containers-roadmap/issues/2570) on GitHub.
For more information about updating your cluster, see [Step 3: Update cluster control plane](update-cluster.md#update-cluster-control-plane).
Before updating your cluster Kubernetes version, you can use the **Upgrade insights** tab of the observability dashboard in the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters). If your cluster has identified issues, review them and make appropriate fixes. The issues include links to Amazon EKS and Kubernetes documentation. After fixing the issue, refresh cluster insights on-demand to fetch the latest insights. If all issues have been resolved, [update your cluster](update-cluster.md).
Amazon EKS returns insights related to Kubernetes version upgrade readiness. Upgrade insights identify possible issues that could impact Kubernetes cluster upgrades. This minimizes the effort that administrators spend preparing for upgrades and increases the reliability of applications on newer Kubernetes versions. Clusters are automatically scanned by Amazon EKS against a list of possible Kubernetes version upgrade impacting issues. Amazon EKS frequently updates the list of insight checks based on reviews of changes made in each Kubernetes version release.
Amazon EKS upgrade insights speed up the testing and verification process for new versions. They also allow cluster administrators and application developers to leverage the newest Kubernetes capabilities by highlighting concerns and offering remediation advice.
### Configuration insights
EKS cluster insights automatically scans Amazon EKS clusters with hybrid nodes to identify configuration issues impairing Kubernetes control plane-to-webhook communication, kubectl commands like exec and logs, and more. Configuration insights surface issues and provide remediation recommendations, accelerating the time to a fully functioning hybrid nodes setup.
## Get started
To see the list of insight checks performed and any relevant issues that Amazon EKS has identified, you can use the AWS Management Console, the AWS CLI, AWS SDKs, and Amazon EKS `ListInsights` API operation. To get started, see [View cluster insights](view-cluster-insights.md).
# View cluster insights
Amazon EKS provides two types of insights: **Configuration insights** and **Upgrade insights**. **Configuration insights** identify misconfigurations in your EKS Hybrid Nodes setup that could impair functionality of your cluster or workloads. **Upgrade insights** identify issues that could impact your ability to upgrade to new versions of Kubernetes.
To see the list of insight checks performed and any relevant issues that Amazon EKS has identified, you can call the look in the AWS Management Console, the AWS CLI, AWS SDKs, and Amazon EKS `ListInsights` API operation.
## View configuration insights (Console)
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. From the cluster list, choose the name of the Amazon EKS cluster for which you want to see the insights.
1. Choose **Monitor cluster**.
1. Choose the **Cluster health** tab.
1. In the **Configuration insights** table, you will see the following columns:
+ **Name** – The check that was performed by Amazon EKS against the cluster.
+ **Insight status** – An insight with a status of `Error` means that there is a misconfiguration that is likely impacting cluster functionality. An insight with a status of `Warning` means that the configuration doesn’t match the documented approach, but that cluster functionality might work if you configured it intentionally. An insight with status of `Passing` means Amazon EKS has not found any issues associated with this insight check in your cluster.
+ **Version** – The applicable version.
+ **Last refresh time** – The time the status of the insight was last refreshed for this cluster.
+ **Description** – Information from the insight check, which includes the alert and recommended actions for remediation.
## View upgrade insights (Console)
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. From the cluster list, choose the name of the Amazon EKS cluster for which you want to see the insights.
1. Choose **Monitor cluster**.
1. Choose the **Upgrade insights** tab.
1. To view the latest data, choose the **Refresh insights** button and wait for the refresh operation to complete.
1. In the **Upgrade insights** table, you will see the following columns:
+ **Name** – The check that was performed by Amazon EKS against the cluster.
+ **Insight status** – An insight with a status of "Error" typically means the impacted Kubernetes version is N\$11 of the current cluster version, while a status of "Warning" means the insight applies to a future Kubernetes version N\$12 or more. An insight with status of "Passing" means Amazon EKS has not found any issues associated with this insight check in your cluster. An insight status of "Unknown" means Amazon EKS is unable to determine if your cluster is impacted by this insight check.
+ **Version** – The Kubernetes version that the insight checked for possible issues.
+ **Last refresh time** – The time the status of the insight was last refreshed for this cluster.
+ **Last transition time** – The time the status of this insight last changed.
+ **Description** – Information from the insight check, which includes the alert and recommended actions for remediation.
## View cluster insights (AWS CLI)
1. To view the latest data, refresh the insights for a specified cluster. Make the following modifications to the command as needed and then run the modified command.
+ Replace *region-code* with the code for your AWS Region.
+ Replace *my-cluster* with the name of your cluster.
```
aws eks start-insights-refresh --region region-code --cluster-name my-cluster
```
1. To track the status of an insights refresh, run the following command. Replace *my-cluster* with the name of your cluster.
```
aws eks describe-insights-refresh --cluster-name my-cluster
```
An example output is as follows.
```
{
"message": "Insights refresh is in progress",
"status": "IN_PROGRESS",
"startedAt": "2025-07-30T13:36:09-07:00"
}
```
1. List the insights for a specified cluster. Make the following modifications to the command as needed and then run the modified command.
+ Replace *region-code* with the code for your AWS Region.
+ Replace *my-cluster* with the name of your cluster.
```
aws eks list-insights --region region-code --cluster-name my-cluster
```
An example output is as follows.
```
{
"insights":
[
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111",
"name": "Deprecated APIs removed in Kubernetes vX.XX",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "X.XX",
"lastRefreshTime": 1734557315.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks for usage of deprecated APIs that are scheduled for removal in Kubernetes vX.XX. Upgrading your cluster before migrating to the updated APIs supported by vX.XX could cause application impact.",
"insightStatus":
{
"status": "PASSING",
"reason": "No deprecated API usage detected within the last 30 days.",
},
},
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLE22222",
"name": "Kubelet version skew",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "X.XX",
"lastRefreshTime": 1734557309.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks for kubelet versions of worker nodes in the cluster to see if upgrade would cause non compliance with supported Kubernetes kubelet version skew policy.",
"insightStatus":
{
"status": "UNKNOWN",
"reason": "Unable to determine status of node kubelet versions.",
},
},
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLE33333",
"name": "Deprecated APIs removed in Kubernetes vX.XX",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "X.XX",
"lastRefreshTime": 1734557315.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks for usage of deprecated APIs that are scheduled for removal in Kubernetes vX.XX. Upgrading your cluster before migrating to the updated APIs supported by vX.XX could cause application impact.",
"insightStatus":
{
"status": "PASSING",
"reason": "No deprecated API usage detected within the last 30 days.",
},
},
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLEaaaaa",
"name": "Cluster health issues",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "X.XX",
"lastRefreshTime": 1734557314.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks for any cluster health issues that prevent successful upgrade to the next Kubernetes version on EKS.",
"insightStatus":
{
"status": "PASSING",
"reason": "No cluster health issues detected.",
},
},
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLEbbbbb",
"name": "EKS add-on version compatibility",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "X.XX",
"lastRefreshTime": 1734557314.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks version of installed EKS add-ons to ensure they are compatible with the next version of Kubernetes. ",
"insightStatus": { "status": "PASSING", "reason": "All installed EKS add-on versions are compatible with next Kubernetes version."},
},
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLEccccc",
"name": "kube-proxy version skew",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "X.XX",
"lastRefreshTime": 1734557314.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks version of kube-proxy in cluster to see if upgrade would cause non compliance with supported Kubernetes kube-proxy version skew policy.",
"insightStatus":
{
"status": "PASSING",
"reason": "kube-proxy versions match the cluster control plane version.",
},
},
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLEddddd",
"name": "Deprecated APIs removed in Kubernetes vX.XX",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "X.XX",
"lastRefreshTime": 1734557315.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks for usage of deprecated APIs that are scheduled for removal in Kubernetes vX.XX. Upgrading your cluster before migrating to the updated APIs supported by vX.XX could cause application impact.",
"insightStatus":
{
"status": "PASSING",
"reason": "No deprecated API usage detected within the last 30 days.",
},
},
],
"nextToken": null,
}
```
1. For descriptive information about an insight, run the following command. Make the following modifications to the command as needed and then run the modified command.
+ Replace *region-code* with the code for your AWS Region.
+ Replace *a1b2c3d4-5678-90ab-cdef-EXAMPLE22222* with an insight ID retrieved from listing the cluster insights.
+ Replace *my-cluster* with the name of your cluster.
```
aws eks describe-insight --region region-code --id a1b2c3d4-5678-90ab-cdef-EXAMPLE22222 --cluster-name my-cluster
```
An example output is as follows.
```
{
"insight":
{
"id": "a1b2c3d4-5678-90ab-cdef-EXAMPLE22222",
"name": "Kubelet version skew",
"category": "UPGRADE_READINESS",
"kubernetesVersion": "1.27",
"lastRefreshTime": 1734557309.000,
"lastTransitionTime": 1734557309.000,
"description": "Checks for kubelet versions of worker nodes in the cluster to see if upgrade would cause non compliance with supported Kubernetes kubelet version skew policy.",
"insightStatus":
{
"status": "UNKNOWN",
"reason": "Unable to determine status of node kubelet versions.",
},
"recommendation": "Upgrade your worker nodes to match the Kubernetes version of your cluster control plane.",
"additionalInfo":
{
"Kubelet version skew policy": "https://kubernetes.io/releases/version-skew-policy/#kubelet",
"Updating a managed node group": "https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html",
},
"resources": [],
"categorySpecificSummary":
{ "deprecationDetails": [], "addonCompatibilityDetails": [] },
},
}
```
# Update existing cluster to new Kubernetes version
**Tip**
[Register](https://aws-experience.com/emea/smb/events/series/get-hands-on-with-amazon-eks?trk=4a9b4147-2490-4c63-bc9f-f8a84b122c8c&sc_channel=el) for upcoming Amazon EKS workshops.
When a new Kubernetes version is available in Amazon EKS, you can update your Amazon EKS cluster to the latest version.
**Important**
Once you upgrade a cluster, you can’t downgrade to a previous version. Before you update to a new Kubernetes version, we recommend that you review the information in [Understand the Kubernetes version lifecycle on EKS](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html) and the update steps in this topic.
New Kubernetes versions sometimes introduce significant changes. Therefore, we recommend that you test the behavior of your applications against a new Kubernetes version before you update your production clusters. You can do this by building a continuous integration workflow to test your application behavior before moving to a new Kubernetes version.
The update process consists of Amazon EKS launching new API server nodes with the updated Kubernetes version to replace the existing ones. Amazon EKS performs standard infrastructure and readiness health checks for network traffic on these new nodes to verify that they’re working as expected. However, once you’ve started the cluster upgrade, you can’t pause or stop it. If any of these checks fail, Amazon EKS reverts the infrastructure deployment, and your cluster remains on the prior Kubernetes version. Running applications aren’t affected, and your cluster is never left in a non-deterministic or unrecoverable state. Amazon EKS regularly backs up all managed clusters, and mechanisms exist to recover clusters if necessary. We’re constantly evaluating and improving our Kubernetes infrastructure management processes.
To update the cluster, Amazon EKS requires up to five available IP addresses from the subnets that you specified when you created your cluster. Amazon EKS creates new cluster elastic network interfaces (network interfaces) in any of the subnets that you specified. The network interfaces may be created in different subnets than your existing network interfaces are in, so make sure that your security group rules allow [required cluster communication](sec-group-reqs.md) for any of the subnets that you specified when you created your cluster. If any of the subnets that you specified when you created the cluster don’t exist, don’t have enough available IP addresses, or don’t have security group rules that allows necessary cluster communication, then the update can fail.
To ensure that the API server endpoint for your cluster is always accessible, Amazon EKS provides a highly available Kubernetes control plane and performs rolling updates of API server instances during update operations. In order to account for changing IP addresses of API server instances supporting your Kubernetes API server endpoint, you must ensure that your API server clients manage reconnects effectively. Recent versions of `kubectl` and the Kubernetes client [libraries](https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#programmatic-access-to-the-api) that are officially supported, perform this reconnect process transparently.
**Note**
To learn more about what goes into a cluster update, see [Best Practices for Cluster Upgrades](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html) in the EKS Best Practices Guide. This resource helps you plan an upgrade, and understand the strategy of upgrading a cluster.
## Considerations for Amazon EKS Auto Mode
+ The compute capability of Amazon EKS Auto Mode controls the Kubernetes version of nodes. After you upgrade the control plane, EKS Auto Mode will begin incrementally updating managed nodes. EKS Auto Mode respects pod disruption budgets.
+ You do not have to manually upgrade the capabilities of Amazon EKS Auto Mode, including the compute autoscaling, block storage, and load balancing capabilities.
## Summary
The high-level summary of the Amazon EKS cluster upgrade process is as follows:
1. Ensure your cluster is in a state that will support an upgrade. This includes checking the Kubernetes APIs used by resources deployed into the cluster, ensuring the cluster is free of any health issues. You should use Amazon EKS upgrade insights when evaluating your cluster’s upgrade readiness.
1. Upgrade the control plane to the next minor version (for example, from 1.34 to 1.35).
1. Upgrade the nodes in the data plane to match that of the control plane.
1. Upgrade any additional applications that run on the cluster (for example, `cluster-autoscaler`).
1. Upgrade the add-ons provided by Amazon EKS, such as those included by default:
+ [Amazon VPC CNI recommended version](managing-vpc-cni.md)
+ [CoreDNS recommended version](managing-coredns.md)
+ [`kube-proxy` recommended version](managing-kube-proxy.md)
1. Upgrade any clients that communicate with the cluster (for example, `kubectl`).
## Step 1: Prepare for upgrade
Compare the Kubernetes version of your cluster control plane to the Kubernetes version of your nodes.
+ Get the Kubernetes version of your cluster control plane.
```
kubectl version
```
+ Get the Kubernetes version of your nodes. This command returns all self-managed and managed Amazon EC2, Fargate, and hybrid nodes. Each Fargate Pod is listed as its own node.
```
kubectl get nodes
```
Before updating your control plane to a new Kubernetes version, make sure that the Kubernetes minor version of both the managed nodes and Fargate nodes in your cluster are the same as your control plane’s version. For example, if your control plane is running version `1.29` and one of your nodes is running version `1.28`, then you must update your nodes to version `1.29` before updating your control plane to 1.30. We also recommend that you update your self-managed nodes and hybrid nodes to the same version as your control plane before updating the control plane. For more information, see [Update a managed node group for your cluster](update-managed-node-group.md), [Update self-managed nodes for your cluster](update-workers.md), and [Upgrade hybrid nodes for your cluster](hybrid-nodes-upgrade.md). If you have Fargate nodes with a minor version lower than the control plane version, first delete the Pod that’s represented by the node. Then update your control plane. Any remaining Pods will update to the new version after you redeploy them.
## Step 2: Review upgrade considerations
Amazon EKS cluster insights automatically scan clusters against a list of potential Kubernetes version upgrade impacting issues such as deprecated Kubernetes API usage. Amazon EKS periodically updates the list of insight checks to perform based on evaluations of changes in the Kubernetes project. Amazon EKS also updates the insight checks list as changes are introduced in the Amazon EKS service along with new versions. For more information, see [Prepare for Kubernetes version upgrades and troubleshoot misconfigurations with cluster insights](cluster-insights.md).
Review the [Deprecated API Migration Guide](https://kubernetes.io/docs/reference/using-api/deprecation-guide/) in the Kubernetes docs.
### Review upgrade insights
Use Amazon EKS upgrade insights to identify issues. For more information, see [View upgrade insights (Console)](view-cluster-insights.md#view-upgrade-insights-console).
### Detailed considerations
+ Because Amazon EKS runs a highly available control plane, you can update only one minor version at a time. For more information about this requirement, see [Kubernetes Version and Version Skew Support Policy](https://kubernetes.io/docs/setup/version-skew-policy/#kube-apiserver). Assume that your current cluster version is version `1.28` and you want to update it to version `1.30`. You must first update your version `1.28` cluster to version `1.29` and then update your version `1.29` cluster to version `1.30`.
+ Review the version skew between the Kubernetes `kube-apiserver` and the `kubelet` on your nodes.
+ Starting from Kubernetes version `1.28`, `kubelet` may be up to three minor versions older than `kube-apiserver`. See [Kubernetes upstream version skew policy](https://kubernetes.io/releases/version-skew-policy/#kubelet).
+ If the `kubelet` on your managed and Fargate nodes is on Kubernetes version `1.25` or newer, you can update your cluster up to three versions ahead without updating the `kubelet` version. For example, if the `kubelet` is on version `1.25`, you can update your Amazon EKS cluster version from `1.25` to `1.26`, to `1.27`, and to `1.28` while the `kubelet` remains on version `1.25`.
+ As a best practice before starting an update, make sure that the `kubelet` on your nodes is at the same Kubernetes version as your control plane.
+ If your cluster is configured with a version of the Amazon VPC CNI plugin for Kubernetes that is earlier than `1.8.0`, then we recommend that you update the plugin to the latest version before updating your cluster. To update the plugin, see [Assign IPs to Pods with the Amazon VPC CNI](managing-vpc-cni.md).
+ You can take a backup of your Amazon EKS cluster, to allow you to restore your Amazon EKS cluster state and persistent storage in the case of failures during the upgrade process. See [Backup your EKS Clusters with AWS Backup](integration-backup.md)
## Step 3: Update cluster control plane
**Important**
Amazon EKS has temporarily rolled back a feature that would require you to use a `--force` flag to upgrade your cluster when there were certain cluster insight issues. For more information, see [Temporary rollback of enforcing upgrade insights on update cluster version](https://github.com/aws/containers-roadmap/issues/2570) on GitHub.
Amazon EKS refreshes a cluster insight 24 hours after the "last refresh time". You can compare the time you addressed an issue to the "last refresh time" of the cluster insight.
Additionally, it can take up to 30 days for the insight status to update after addressing deprecated API usage. Upgrade insights always looks for deprecated API usage over a rolling 30 day window.
You can submit the request to upgrade your EKS control plane version using:
+ [eksctl](#step3-eksctl)
+ [the AWS console](#step3-console)
+ [the AWS CLI](#step3-cli)
### Update cluster - eksctl
This procedure requires `eksctl` version `0.215.0` or later. You can check your version with the following command:
```
eksctl version
```
For instructions on how to install and update `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
Update the Kubernetes version of your Amazon EKS control plane. Replace `` with your cluster name. Replace `` with the Amazon EKS supported version number that you want to update your cluster to. For a list of supported version numbers, see [Amazon EKS supported versions](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
```
eksctl upgrade cluster --name --version --approve
```
The update takes several minutes to complete.
Continue to [Step 4: Update cluster components](#step4).
### Update cluster - AWS console
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose **Upgrade now** for a cluster you wish to upgrade.
1. Select the version to update your cluster to and choose **Upgrade**.
1. The update takes several minutes to complete. Continue to [Step 4: Update cluster components](#step4).
### Update cluster - AWS CLI
1. Verify that the AWS CLI is installed and that you are logged in. For more information, see [Installing or updating to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
1. Update your Amazon EKS cluster with the following AWS CLI command. Replace `` and `` of the cluster you want to upgrade. Replace `` with the Amazon EKS supported version number that you want to update your cluster to. For a list of supported version numbers, see [Amazon EKS supported versions](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
```
aws eks update-cluster-version --name \
--kubernetes-version --region
```
An example output is as follows.
```
{
"update": {
"id": "",
"status": "InProgress",
"type": "VersionUpdate",
"params": [
{
"type": "Version",
"value": ""
},
{
"type": "PlatformVersion",
"value": "eks.1"
}
],
[...]
"errors": []
}
```
1. The update takes several minutes to complete. Monitor the status of your cluster update with the following command. In addition to using the same `` and ``, use the `` that the previous command returned.
```
aws eks describe-update --name \
--region --update-id
```
When a `Successful` status is displayed, the update is complete.
1. Continue to [Step 4: Update cluster components](#step4).
## Step 4: Update cluster components
1. After your cluster update is complete, update your nodes to the same Kubernetes minor version as your updated cluster. For more information, see [Update self-managed nodes for your cluster](update-workers.md), [Update a managed node group for your cluster](update-managed-node-group.md), and [Upgrade hybrid nodes for your cluster](hybrid-nodes-upgrade.md). Any new Pods that are launched on Fargate have a `kubelet` version that matches your cluster version. Existing Fargate Pods aren’t changed.
1. (Optional) If you deployed the Kubernetes Cluster Autoscaler to your cluster before updating the cluster, update the Cluster Autoscaler to the latest version that matches the Kubernetes major and minor version that you updated to.
1. Open the Cluster Autoscaler [releases](https://github.com/kubernetes/autoscaler/releases) page in a web browser and find the latest Cluster Autoscaler version that matches your cluster’s Kubernetes major and minor version. For example, if your cluster’s Kubernetes version is `1.30` find the latest Cluster Autoscaler release that begins with `1.30`. Record the semantic version number (`1.30.n`, for example) for that release to use in the next step.
1. Set the Cluster Autoscaler image tag to the version that you recorded in the previous step with the following command. If necessary, replace `X.XX.X` with your own value.
```
kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=registry.k8s.io/autoscaling/cluster-autoscaler:vX.XX.X
```
1. (Clusters with GPU nodes only) If your cluster has node groups with GPU support (for example, `p3.2xlarge`), you must update the [NVIDIA device plugin for Kubernetes](https://github.com/NVIDIA/k8s-device-plugin)DaemonSet on your cluster. Replace `` with your desired [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin/releases) version before running the following command.
```
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin//deployments/static/nvidia-device-plugin.yml
```
1. Update the Amazon VPC CNI plugin for Kubernetes, CoreDNS, and `kube-proxy` add-ons. We recommend updating the add-ons to the minimum versions listed in [Service account tokens](service-accounts.md#boundserviceaccounttoken-validated-add-on-versions).
+ If you are using Amazon EKS add-ons, select **Clusters** in the Amazon EKS console, then select the name of the cluster that you updated in the left navigation pane. Notifications appear in the console. They inform you that a new version is available for each add-on that has an available update. To update an add-on, select the **Add-ons** tab. In one of the boxes for an add-on that has an update available, select **Update now**, select an available version, and then select **Update**.
+ Alternately, you can use the AWS CLI or `eksctl` to update add-ons. For more information, see [Update an Amazon EKS add-on](updating-an-add-on.md).
1. If necessary, update your version of `kubectl`. You must use a `kubectl` version that is within one minor version difference of your Amazon EKS cluster control plane.
## Downgrade the Kubernetes version for an Amazon EKS cluster
You cannot downgrade the Kubernetes of an Amazon EKS cluster. Instead, create a new cluster on a previous Amazon EKS version and migrate the workloads.
# Delete a cluster
When you’re done using an Amazon EKS cluster, you should delete the resources associated with it so that you don’t incur any unnecessary costs.
You can delete a cluster with `eksctl`, the AWS Management Console, or the AWS CLI.
## Considerations
+ If you receive an error because the cluster creator has been removed, see [this article](https://aws.amazon.com/premiumsupport/knowledge-center/eks-api-server-unauthorized-error) to resolve.
+ Amazon Managed Service for Prometheus resources are outside of the cluster lifecycle and need to be maintained independent of the cluster. When you delete your cluster, make sure to also delete any applicable scrapers to stop applicable costs. For more information, see [Find and delete scrapers](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-collector-how-to.html#AMP-collector-list-delete) in the *Amazon Managed Service for Prometheus User Guide*.
+ To remove a connected cluster, see [Deregister a Kubernetes cluster from the Amazon EKS console](deregister-connected-cluster.md)
+ Before you can delete a cluster, make sure deletion protection is disabled for your cluster.
### Considerations for EKS Auto Mode
+ Any EKS Auto Mode Nodes will be deleted, including the EC2 managed instances
+ All load balancers will be deleted
For more information, see [Disable EKS Auto Mode](auto-disable.md).
## Prerequisite steps
The following are steps that you must first perform before you can delete a cluster. These steps apply regardless of the method that you use to delete your cluster.
1. List all services running in your cluster.
```
kubectl get svc --all-namespaces
```
1. Delete any services that have an associated `EXTERNAL-IP` value. These services are fronted by an Elastic Load Balancing load balancer, and you must delete them in Kubernetes to allow the load balancer and associated resources to be properly released. Replace *service-name* with the name of each service listed as described.
```
kubectl delete svc service-name
```
1. Delete any ingress resources as well. If you don’t delete the ingress resources, the application load balancer remains even if you deleted the cluster. Replace *ingress-name* with the name of your ingress resources.
```
kubectl get ingress --all-namespaces
```
```
kubectl delete ing ingress-name
```
## Delete cluster (eksctl)
This procedure requires `eksctl` version `0.215.0` or later. You can check your version with the following command:
```
eksctl version
```
For instructions on how to install or upgrade `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation.
1. Go through the [prerequisite steps](#prerequisite-steps). After doing so, delete your cluster and its associated nodes with the following command, replacing *prod* with your cluster name.
```
eksctl delete cluster --name prod
```
Output:
```
[ℹ] using region region-code
[ℹ] deleting EKS cluster "prod"
[ℹ] will delete stack "eksctl-prod-nodegroup-standard-nodes"
[ℹ] waiting for stack "eksctl-prod-nodegroup-standard-nodes" to get deleted
[ℹ] will delete stack "eksctl-prod-cluster"
[✔] the following EKS cluster resource(s) for "prod" will be deleted: cluster. If in doubt, check CloudFormation console
```
## Delete cluster (AWS console)
1. Go through the [prerequisite steps](#prerequisite-steps). After doing so, delete all node groups and Fargate profiles.
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. In the left navigation pane, choose Amazon EKS **Clusters**, and then in the tabbed list of clusters, choose the name of the cluster that you want to delete.
1. Choose the **Compute** tab and choose a node group to delete. Choose **Delete**, enter the name of the node group, and then choose **Delete**. Delete all node groups in the cluster.
**Note**
The node groups listed are [managed node groups](managed-node-groups.md) only.
1. Choose a **Fargate Profile** to delete, select **Delete**, enter the name of the profile, and then choose **Delete**. Delete all Fargate profiles in the cluster.
1. Delete all [self-managed node AWS CloudFormation stacks](https://docs.aws.amazon.com/eks/latest/userguide/worker).
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Choose the node stack to delete, and then choose **Delete**.
1. In the **Delete stack** confirmation dialog box, choose **Delete stack**. Delete all self-managed node stacks in the cluster.
1. Delete the cluster.
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. choose the cluster to delete and choose **Delete**.
1. On the delete cluster confirmation screen, choose **Delete**.
1. (Optional) Delete the VPC AWS CloudFormation stack.
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Select the VPC stack to delete, and then choose **Delete**.
1. In the **Delete stack** confirmation dialog box, choose **Delete stack**.
## Delete cluster (AWS CLI)
1. Go through the [prerequisite steps](#prerequisite-steps). After doing so, delete all node groups and Fargate profiles.
1. List the node groups in your cluster with the following command.
```
aws eks list-nodegroups --cluster-name my-cluster
```
**Note**
The node groups listed are [managed node groups](managed-node-groups.md) only.
1. Delete each node group with the following command. Delete all node groups in the cluster.
```
aws eks delete-nodegroup --nodegroup-name my-nodegroup --cluster-name my-cluster
```
1. List the Fargate profiles in your cluster with the following command.
```
aws eks list-fargate-profiles --cluster-name my-cluster
```
1. Delete each Fargate profile with the following command. Delete all Fargate profiles in the cluster.
```
aws eks delete-fargate-profile --fargate-profile-name my-fargate-profile --cluster-name my-cluster
```
1. Delete all [self-managed node AWS CloudFormation stacks](https://docs.aws.amazon.com/eks/latest/userguide/worker).
1. List your available AWS CloudFormation stacks with the following command. Find the node template name in the resulting output.
```
aws cloudformation list-stacks --query "StackSummaries[].StackName"
```
1. Delete each node stack with the following command, replacing *node-stack* with your node stack name. Delete all self-managed node stacks in the cluster.
```
aws cloudformation delete-stack --stack-name node-stack
```
1. Delete the cluster with the following command, replacing *my-cluster* with your cluster name.
```
aws eks delete-cluster --name my-cluster
```
1. (Optional) Delete the VPC AWS CloudFormation stack.
1. List your available AWS CloudFormation stacks with the following command. Find the VPC template name in the resulting output.
```
aws cloudformation list-stacks --query "StackSummaries[].StackName"
```
1. Delete the VPC stack with the following command, replacing *my-vpc-stack* with your VPC stack name.
```
aws cloudformation delete-stack --stack-name my-vpc-stack
```
# Protect EKS clusters from accidental deletion
Accidentally deleting an EKS cluster may impair Kubernetes cluster operations.
You can now protect EKS clusters from accidental deletion. If you enable deletion protection on a cluster, you must first disable deletion protection before you can delete the cluster.
The purpose of deletion protection is to prevent accidents. You should carefully restrict who is authorized to delete clusters.
If you try to delete an active cluster that has deletion protection turned on, you will receive a `InvalidRequestException` .
**Important**
If you enable deletion protection on a cluster, you must have **both** the UpdateClusterConfig and DeleteCluster IAM permissions to first remove the deletion protection, and finally delete the cluster.
**Note**
If the cluster state is creating, failed, or deleting, you can delete the cluster even if deletion protection is turned on.
## To enable deletion protection for an existing cluster
You can only run this on a cluster in the active status.
```
aws eks update-cluster-config --name --region --deletion-protection
```
## To disable deletion protection for an existing cluster
```
aws eks update-cluster-config --name --region --no-deletion-protection
```
# Cluster API server endpoint
This topic helps you to enable private access for your Amazon EKS cluster’s Kubernetes API server endpoint and limit, or completely disable, public access from the internet.
When you create a new cluster, Amazon EKS creates an endpoint for the managed Kubernetes API server that you use to communicate with your cluster (using Kubernetes management tools such as `kubectl`). By default, this API server endpoint is public to the internet, and access to the API server is secured using a combination of AWS Identity and Access Management (IAM) and native Kubernetes [Role Based Access Control](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) (RBAC). This endpoint is known as the *cluster public endpoint*. Also there is a *cluster private endpoint*. For more information about the cluster private endpoint, see the following section [Cluster private endpoint](#cluster-endpoint-private).
## `IPv6` cluster endpoint format
EKS creates a unique dual-stack endpoint in the following format for new `IPv6` clusters that are made after October 2024. An *IPv6 cluster* is a cluster that you select `IPv6` in the IP family (`ipFamily`) setting of the cluster.
**Example**
EKS cluster public/private endpoint: `eks-cluster.region.api.aws`
EKS cluster public/private endpoint: `eks-cluster.region.api.aws`
EKS cluster public/private endpoint: `eks-cluster---region---api.amazonwebservices.com.rproxy.goskope.com.cn`
**Note**
The dual-stack cluster endpoint was introduced in October 2024. For more information about `IPv6` clusters, see [Learn about IPv6 addresses to clusters, Pods, and services](cni-ipv6.md). Clusters made before October 2024, use following endpoint format instead.
## `IPv4` cluster endpoint format
EKS creates a unique endpoint in the following format for each cluster that select `IPv4` in the IP family (ipFamily) setting of the cluster:
**Example**
EKS cluster public/private endpoint `eks-cluster.region.eks.amazonaws.com`
EKS cluster public/private endpoint `eks-cluster.region.eks.amazonaws.com`
EKS cluster public/private endpoint `eks-cluster---region.amazonwebservices.com.rproxy.goskope.com.cn`
**Note**
Before October 2024, `IPv6` clusters used this endpoint format also. For those clusters, both the public endpoint and the private endpoint have only `IPv4` addresses resolve from this endpoint.
## Cluster private endpoint
You can enable private access to the Kubernetes API server so that all communication between your nodes and the API server stays within your VPC. You can limit the IP addresses that can access your API server from the internet, or completely disable internet access to the API server.
**Note**
Because this endpoint is for the Kubernetes API server and not a traditional AWS PrivateLink endpoint for communicating with an AWS API, it doesn’t appear as an endpoint in the Amazon VPC console.
When you enable endpoint private access for your cluster, Amazon EKS creates a Route 53 private hosted zone on your behalf and associates it with your cluster’s VPC. This private hosted zone is managed by Amazon EKS, and it doesn’t appear in your account’s Route 53 resources. In order for the private hosted zone to properly route traffic to your API server, your VPC must have `enableDnsHostnames` and `enableDnsSupport` set to `true`, and the DHCP options set for your VPC must include `AmazonProvidedDNS` in its domain name servers list. For more information, see [Updating DNS support for your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating) in the *Amazon VPC User Guide*.
You can define your API server endpoint access requirements when you create a new cluster, and you can update the API server endpoint access for a cluster at any time.
## Modifying cluster endpoint access
Use the procedures in this section to modify the endpoint access for an existing cluster. The following table shows the supported API server endpoint access combinations and their associated behavior.
| Endpoint public access | Endpoint private access | Behavior |
| --- | --- | --- |
| Enabled | Disabled | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) |
| Enabled | Enabled | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) |
| Disabled | Enabled | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) |
**Endpoint access controls**
Note that each of the following methods to control endpoint access only affect the respective endpoint.
*Cluster security group*
The cluster security group controls two types of connections: connections to the *kubelet API* and the private endpoint. The connections to the `kubelet` API are used in the `kubectl attach`, `kubectl cp`, `kubectl exec`, `kubectl logs`, and `kubectl port-forward` commands. The cluster security group doesn’t affect the public endpoint.
*Public access CIDRs*
The *public access CIDRs* control access to the public endpoint by a list of CIDR blocks. Note that the public access CIDRs don’t affect the private endpoint. The public access CIDRs behave differently on the `IPv6` clusters and `IPv4` clusters depending on the date they were created, which the following describes:
**CIDR blocks in the public endpoint (`IPv6` cluster)**
You can add `IPv6` and `IPv4` CIDR blocks to the public endpoint of an `IPv6` cluster, because the public endpoint is dual-stack. This only applies to new clusters with the `ipFamily` set to `IPv6` that you made in October 2024 or later. You can identify these clusters by the new endpoint domain name `api.aws`.
**CIDR blocks in the public endpoint (`IPv4` cluster)**
You can add `IPv4` CIDR blocks to the public endpoint of an `IPv4` cluster. You can’t add `IPv6` CIDR blocks to the public endpoint of an `IPv4` cluster. If you try, EKS returns the following error message: `The following CIDRs are invalid in publicAccessCidrs`
**CIDR blocks in the public endpoint (`IPv6` cluster made before October 2024)**
You can add `IPv4` CIDR blocks to the public endpoint of the old `IPv6` clusters that you made before October 2024. You can identify these clusters by the `eks.amazonaws.com` endpoint. You can’t add `IPv6` CIDR blocks to the public endpoint of these old `IPv6` clusters that you made before October 2024. If you try, EKS returns the following error message: `The following CIDRs are invalid in publicAccessCidrs`
## Accessing a private only API server
If you have disabled public access for your cluster’s Kubernetes API server endpoint, you can only access the API server from within your VPC or a [connected network](https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/introduction.html). Here are a few possible ways to access the Kubernetes API server endpoint:
**Connected network**
Connect your network to the VPC with an [AWS transit gateway](https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html) or other [connectivity](https://docs.aws.amazon.com/aws-technical-content/latest/aws-vpc-connectivity-options/introduction.html) option and then use a computer in the connected network. You must ensure that your Amazon EKS control plane security group contains rules to allow ingress traffic on port 443 from your connected network.
**Amazon EC2 bastion host**
You can launch an Amazon EC2 instance into a public subnet in your cluster’s VPC and then log in via SSH into that instance to run `kubectl` commands. For more information, see [Linux bastion hosts on AWS](https://aws.amazon.com/quickstart/architecture/linux-bastion/). You must ensure that your Amazon EKS control plane security group contains rules to allow ingress traffic on port 443 from your bastion host. For more information, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md).
When you configure `kubectl` for your bastion host, be sure to use AWS credentials that are already mapped to your cluster’s RBAC configuration, or add the [IAM principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html#iam-term-principal) that your bastion will use to the RBAC configuration before you remove endpoint public access. For more information, see [Grant IAM users and roles access to Kubernetes APIs](grant-k8s-access.md) and [Unauthorized or access denied (`kubectl`)](troubleshooting.md#unauthorized).
** AWS Cloud9 IDE**
AWS Cloud9 is a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser. You can create an AWS Cloud9 IDE in your cluster’s VPC and use the IDE to communicate with your cluster. For more information, see [Creating an environment in AWS Cloud9](https://docs.aws.amazon.com/cloud9/latest/user-guide/create-environment.html). You must ensure that your Amazon EKS control plane security group contains rules to allow ingress traffic on port 443 from your IDE security group. For more information, see [View Amazon EKS security group requirements for clusters](sec-group-reqs.md).
When you configure `kubectl` for your AWS Cloud9 IDE, be sure to use AWS credentials that are already mapped to your cluster’s RBAC configuration, or add the IAM principal that your IDE will use to the RBAC configuration before you remove endpoint public access. For more information, see [Grant IAM users and roles access to Kubernetes APIs](grant-k8s-access.md) and [Unauthorized or access denied (`kubectl`)](troubleshooting.md#unauthorized).
📝 [Edit this page on GitHub](https://github.com/search?q=repo%3Aawsdocs%2Famazon-eks-user-guide+%5B%23cluster-endpoint%5D&type=code)
# Configure network access to cluster API server endpoint
You can modify your cluster API server endpoint access using the AWS Management Console or AWS CLI in the following sections.
## Configure endpoint access - AWS console
1. Open the [Amazon EKS console](https://console.aws.amazon.com/eks/home#/clusters).
1. Choose the name of the cluster to display your cluster information.
1. Choose the **Networking** tab and choose **Manage endpoint access**.
1. For **Private access**, choose whether to enable or disable private access for your cluster’s Kubernetes API server endpoint. If you enable private access, Kubernetes API requests that originate from within your cluster’s VPC use the private VPC endpoint. You must enable private access to disable public access.
1. For **Public access**, choose whether to enable or disable public access for your cluster’s Kubernetes API server endpoint. If you disable public access, your cluster’s Kubernetes API server can only receive requests from within the cluster VPC.
1. (Optional) If you’ve enabled **Public access**, you can specify which addresses from the internet can communicate to the public endpoint. Select **Advanced Settings**. Enter a CIDR block, such as *203.0.113.5/32*. The block cannot include [reserved addresses](https://en.wikipedia.org/wiki/Reserved_IP_addresses). You can enter additional blocks by selecting **Add Source**. There is a maximum number of CIDR blocks that you can specify. For more information, see [View and manage Amazon EKS and Fargate service quotas](service-quotas.md). If you specify no blocks, then the public API server endpoint receives requests from all IP addresses for both `IPv4` (`0.0.0.0/0`) and additionally `IPv6` (`::/0`) for dual-stack `IPv6` cluster. If you restrict access to your public endpoint using CIDR blocks, we recommend that you also enable private endpoint access so that nodes and Fargate Pods (if you use them) can communicate with the cluster. Without the private endpoint enabled, your public access endpoint CIDR sources must include the egress sources from your VPC. For example, if you have a node in a private subnet that communicates to the internet through a NAT Gateway, you will need to add the outbound IP address of the NAT gateway as part of an allowed CIDR block on your public endpoint.
1. Choose **Update** to finish.
## Configure endpoint access - AWS CLI
Complete the following steps using the AWS CLI version `1.27.160` or later. You can check your current version with `aws --version`. To install or upgrade the AWS CLI, see [Installing the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html).
1. Update your cluster API server endpoint access with the following AWS CLI command. Substitute your cluster name and desired endpoint access values. If you set `endpointPublicAccess=true`, then you can (optionally) enter single CIDR block, or a comma-separated list of CIDR blocks for `publicAccessCidrs`. The blocks cannot include [reserved addresses](https://en.wikipedia.org/wiki/Reserved_IP_addresses). If you specify CIDR blocks, then the public API server endpoint will only receive requests from the listed blocks. There is a maximum number of CIDR blocks that you can specify. For more information, see [View and manage Amazon EKS and Fargate service quotas](service-quotas.md). If you restrict access to your public endpoint using CIDR blocks, it is recommended that you also enable private endpoint access so that nodes and Fargate Pods (if you use them) can communicate with the cluster. Without the private endpoint enabled, your public access endpoint CIDR sources must include the egress sources from your VPC. For example, if you have a node in a private subnet that communicates to the internet through a NAT Gateway, you will need to add the outbound IP address of the NAT gateway as part of an allowed CIDR block on your public endpoint. If you specify no CIDR blocks, then the public API server endpoint receives requests from all (0.0.0.0/0) IP addresses and additionally `IPv6` (`::/0`) for dual-stack `IPv6` cluster.
**Note**
The following command enables private access and public access from a single IP address for the API server endpoint. Replace *203.0.113.5/32* with a single CIDR block, or a comma-separated list of CIDR blocks that you want to restrict network access to.
```
aws eks update-cluster-config \
--region region-code \
--name my-cluster \
--resources-vpc-config endpointPublicAccess=true,publicAccessCidrs="203.0.113.5/32",endpointPrivateAccess=true
```
An example output is as follows.
```
{
"update": {
"id": "e6f0905f-a5d4-4a2a-8c49-EXAMPLE00000",
"status": "InProgress",
"type": "EndpointAccessUpdate",
"params": [
{
"type": "EndpointPublicAccess",
"value": "true"
},
{
"type": "EndpointPrivateAccess",
"value": "true"
},
{
"type": "publicAccessCidrs",
"value": "[\"203.0.113.5/32\"]"
}
],
"createdAt": 1576874258.137,
"errors": []
}
}
```
1. Monitor the status of your endpoint access update with the following command, using the cluster name and update ID that was returned by the previous command. Your update is complete when the status is shown as `Successful`.
```
aws eks describe-update \
--region region-code \
--name my-cluster \
--update-id e6f0905f-a5d4-4a2a-8c49-EXAMPLE00000
```
An example output is as follows.
```
{
"update": {
"id": "e6f0905f-a5d4-4a2a-8c49-EXAMPLE00000",
"status": "Successful",
"type": "EndpointAccessUpdate",
"params": [
{
"type": "EndpointPublicAccess",
"value": "true"
},
{
"type": "EndpointPrivateAccess",
"value": "true"
},
{
"type": "publicAccessCidrs",
"value": "[\"203.0.113.5/32\"]"
}
],
"createdAt": 1576874258.137,
"errors": []
}
}
```
📝 [Edit this page on GitHub](https://github.com/search?q=repo%3Aawsdocs%2Famazon-eks-user-guide+%5B%23config-cluster-endpoint%5D&type=code)
# Deploy Windows nodes on EKS clusters
Learn how to enable and manage Windows support for your Amazon EKS cluster to run Windows containers alongside Linux containers.
## Considerations
Before deploying Windows nodes, be aware of the following considerations.
+ EKS Auto Mode does not support Windows nodes
+ You can use host networking on Windows nodes using `HostProcess` Pods. For more information, see [Create a Windows HostProcessPod](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/) in the Kubernetes documentation.
+ Amazon EKS clusters must contain one or more Linux or Fargate nodes to run core system Pods that only run on Linux, such as CoreDNS.
+ The `kubelet` and `kube-proxy` event logs are redirected to the `EKS Windows` Event Log and are set to a 200 MB limit.
+ You can’t use [Assign security groups to individual pods](security-groups-for-pods.md) with Pods running on Windows nodes.
+ You can’t use [custom networking](cni-custom-network.md) with Windows nodes.
+ You can’t use `IPv6` with Windows nodes.
+ Windows nodes support one elastic network interface per node. By default, the number of Pods that you can run per Windows node is equal to the number of IP addresses available per elastic network interface for the node’s instance type, minus one. For more information, see [IP addresses per network interface per instance type](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/using-eni.html#AvailableIpPerENI) in the *Amazon EC2 User Guide*.
+ In an Amazon EKS cluster, a single service with a load balancer can support up to 1024 back-end Pods. Each Pod has its own unique IP address. The previous limit of 64 Pods is no longer the case, after [a Windows Server update](https://github.com/microsoft/Windows-Containers/issues/93) starting with [OS Build 17763.2746](https://support.microsoft.com/en-us/topic/march-22-2022-kb5011551-os-build-17763-2746-preview-690a59cd-059e-40f4-87e8-e9139cc65de4).
+ Windows containers aren’t supported for Amazon EKS Pods on Fargate.
+ You can’t use Amazon EKS Hybrid Nodes with Windows as the operating system for the host.
+ You can’t retrieve logs from the `vpc-resource-controller` Pod. You previously could when you deployed the controller to the data plane.
+ There is a cool down period before an `IPv4` address is assigned to a new Pod. This prevents traffic from flowing to an older Pod with the same `IPv4` address due to stale `kube-proxy` rules.
+ The source for the controller is managed on GitHub. To contribute to, or file issues against the controller, visit the [project](https://github.com/aws/amazon-vpc-resource-controller-k8s) on GitHub.
+ When specifying a custom AMI ID for Windows managed node groups, add `eks:kube-proxy-windows` to your AWS IAM Authenticator configuration map. For more information, see [Limits and conditions when specifying an AMI ID](launch-templates.md#mng-ami-id-conditions).
+ If preserving your available IPv4 addresses is crucial for your subnet, refer to [EKS Best Practices Guide - Windows Networking IP Address Management](https://aws.github.io/aws-eks-best-practices/windows/docs/networking/#ip-address-management) for guidance.
+ Considerations for EKS Access Entries
+ Access Entries for use with Windows nodes need the type of `EC2_WINDOWS`. For more information, see [Create access entries](creating-access-entries.md).
To create an access entry for a Windows node:
```
aws eks create-access-entry --cluster-name my-cluster --principal-arn arn:aws:iam::111122223333:role/ --type EC2_Windows
```
## Prerequisites
+ An existing cluster.
+ Your cluster must have at least one (we recommend at least two) Linux node or Fargate Pod to run CoreDNS. If you enable legacy Windows support, you must use a Linux node (you can’t use a Fargate Pod) to run CoreDNS.
+ An existing [Amazon EKS cluster IAM role](cluster-iam-role.md).
## Enable Windows support
1. If you don’t have Amazon Linux nodes in your cluster and use security groups for Pods, skip to the next step. Otherwise, confirm that the `AmazonEKSVPCResourceController` managed policy is attached to your [cluster role](cluster-iam-role.md). Replace *eksClusterRole* with your cluster role name.
```
aws iam list-attached-role-policies --role-name eksClusterRole
```
An example output is as follows.
```
{
"AttachedPolicies": [
{
"PolicyName": "AmazonEKSClusterPolicy",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
},
{
"PolicyName": "AmazonEKSVPCResourceController",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
}
]
}
```
If the policy is attached, as it is in the previous output, skip the next step.
1. Attach the ** [AmazonEKSVPCResourceController](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonEKSVPCResourceController.html) ** managed policy to your [Amazon EKS cluster IAM role](cluster-iam-role.md). Replace *eksClusterRole* with your cluster role name.
```
aws iam attach-role-policy \
--role-name eksClusterRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController
```
1. Update the VPC CNI ConfigMap to enable Windows IPAM. Note, if the VPC CNI is installed on your cluster using a Helm chart or as an Amazon EKS Add-on you may not be able to directly modify the ConfigMap. For information on configuring Amazon EKS Add-ons, see [Determine fields you can customize for Amazon EKS add-ons](kubernetes-field-management.md).
1. Create a file named *vpc-resource-controller-configmap.yaml* with the following contents.
```
apiVersion: v1
kind: ConfigMap
metadata:
name: amazon-vpc-cni
namespace: kube-system
data:
enable-windows-ipam: "true"
```
1. Apply the `ConfigMap` to your cluster.
```
kubectl apply -f vpc-resource-controller-configmap.yaml
```
1. If your cluster has the authentication mode set to enable the `aws-auth` configmap:
+ Verify that your `aws-auth` `ConfigMap` contains a mapping for the instance role of the Windows node to include the `eks:kube-proxy-windows` RBAC permission group. You can verify by running the following command.
```
kubectl get configmap aws-auth -n kube-system -o yaml
```
An example output is as follows.
```
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
- eks:kube-proxy-windows # This group is required for Windows DNS resolution to work
rolearn: arn:aws:iam::111122223333:role/eksNodeRole
username: system:node:{{EC2PrivateDNSName}}
[...]
```
You should see `eks:kube-proxy-windows` listed under groups. If the group isn’t specified, you need to update your `ConfigMap` or create it to include the required group. For more information about the `aws-auth` `ConfigMap`, see [Apply the `aws-auth` `ConfigMap` to your cluster](auth-configmap.md#aws-auth-configmap).
1. If your cluster has the authentication mode set to disable the `aws-auth` configmap, then you can use EKS Access Entries. Create a new node role for use with Windows instances, and EKS will automatically create an access entry of type `EC2_WINDOWS`.
## Deploy Windows Pods
When you deploy Pods to your cluster, you need to specify the operating system that they use if you’re running a mixture of node types.
For Linux Pods, use the following node selector text in your manifests.
```
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: amd64
```
For Windows Pods, use the following node selector text in your manifests.
```
nodeSelector:
kubernetes.io/os: windows
kubernetes.io/arch: amd64
```
You can deploy a [sample application](sample-deployment.md) to see the node selectors in use.
## Support higher Pod density on Windows nodes
In Amazon EKS, each Pod is allocated an `IPv4` address from your VPC. Due to this, the number of Pods that you can deploy to a node is constrained by the available IP addresses, even if there are sufficient resources to run more Pods on the node. Since only one elastic network interface is supported by a Windows node, by default, the maximum number of available IP addresses on a Windows node is equal to:
```
Number of private IPv4 addresses for each interface on the node - 1
```
One IP address is used as the primary IP address of the network interface, so it can’t be allocated to Pods.
You can enable higher Pod density on Windows nodes by enabling IP prefix delegation. This feature enables you to assign a `/28` `IPv4` prefix to the primary network interface, instead of assigning secondary `IPv4` addresses. Assigning an IP prefix increases the maximum available `IPv4` addresses on the node to:
```
(Number of private IPv4 addresses assigned to the interface attached to the node - 1) * 16
```
With this significantly larger number of available IP addresses, available IP addresses shouldn’t limit your ability to scale the number of Pods on your nodes. For more information, see [Assign more IP addresses to Amazon EKS nodes with prefixes](cni-increase-ip-addresses.md).
# Disable Windows support
1. If your cluster contains Amazon Linux nodes and you use [security groups for Pods](security-groups-for-pods.md) with them, then skip this step.
Remove the `AmazonVPCResourceController` managed IAM policy from your [cluster role](cluster-iam-role.md). Replace *eksClusterRole* with the name of your cluster role.
```
aws iam detach-role-policy \
--role-name eksClusterRole \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController
```
1. Disable Windows IPAM in the `amazon-vpc-cni` ConfigMap.
```
kubectl patch configmap/amazon-vpc-cni \
-n kube-system \
--type merge \
-p '{"data":{"enable-windows-ipam":"false"}}'
```
# Deploy private clusters with limited internet access
This topic describes how to deploy an Amazon EKS cluster that is deployed on the AWS Cloud, but doesn’t have outbound internet access. If you have a local cluster on AWS Outposts, see [Create Amazon Linux nodes on AWS Outposts](eks-outposts-self-managed-nodes.md), instead of this topic.
If you’re not familiar with Amazon EKS networking, see [De-mystifying cluster networking for Amazon EKS worker nodes](https://aws.amazon.com/blogs/containers/de-mystifying-cluster-networking-for-amazon-eks-worker-nodes). If your cluster doesn’t have outbound internet access, then it must meet the following requirements:
## Cluster architecture requirements
+ Your cluster must pull images from a container registry that’s in your VPC. You can create an Amazon Elastic Container Registry in your VPC and copy container images to it for your nodes to pull from. For more information, see [Copy a container image from one repository to another repository](copy-image-to-repository.md).
+ Your cluster must have endpoint private access enabled. This is required for nodes to register with the cluster endpoint. Endpoint public access is optional. For more information, see [Cluster API server endpoint](cluster-endpoint.md).
## Node requirements
+ Self-managed Linux and Windows nodes must include the following bootstrap arguments before they’re launched. These arguments bypass Amazon EKS introspection and don’t require access to the Amazon EKS API from within the VPC.
1. Determine the value of your cluster’s endpoint with the following command. Replace *my-cluster* with the name of your cluster.
```
aws eks describe-cluster --name my-cluster --query cluster.endpoint --output text
```
An example output is as follows.
```
https://EXAMPLE108C897D9B2F1B21D5EXAMPLE.sk1.region-code.eks.amazonaws.com
```
1. Determine the value of your cluster’s certificate authority with the following command. Replace *my-cluster* with the name of your cluster.
```
aws eks describe-cluster --name my-cluster --query cluster.certificateAuthority --output text
```
The returned output is a long string.
1. Replace the values of `apiServerEndpoint` and `certificateAuthority` in the NodeConfig object with the values returned in the output from the previous commands. For more information about specifying bootstrap arguments when launching self-managed Amazon Linux 2023 nodes, see [Create self-managed Amazon Linux nodes](launch-workers.md) and [Create self-managed Microsoft Windows nodes](launch-windows-workers.md).
+ For Linux nodes:
```
---
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"
--BOUNDARY
Content-Type: application/node.eks.aws
---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name: my-cluster
apiServerEndpoint: [.replaceable]https://EXAMPLE108C897D9B2F1B21D5EXAMPLE.sk1.region-code.eks.amazonaws.com
certificateAuthority: [.replaceable]Y2VydGlmaWNhdGVBdXRob3JpdHk=
...
```
For additional arguments, see the [bootstrap script](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh) on GitHub.
+ For Windows nodes:
**Note**
If you’re using custom service CIDR, then you need to specify it using the `-ServiceCIDR` parameter. Otherwise, the DNS resolution for Pods in the cluster will fail.
```
-APIServerEndpoint cluster-endpoint -Base64ClusterCA certificate-authority
```
For additional arguments, see [Bootstrap script configuration parameters](eks-optimized-windows-ami.md#bootstrap-script-configuration-parameters).
+ Your cluster’s `aws-auth` `ConfigMap` must be created from within your VPC. For more information about creating and adding entries to the `aws-auth` `ConfigMap`, enter `eksctl create iamidentitymapping --help` in your terminal. If the `ConfigMap` doesn’t exist on your server, `eksctl` will create it when you use the command to add an identity mapping.
## Pod requirements
+ **Pod Identity** - Pods configured with EKS Pod Identity acquire credentials from the EKS Auth API. If there is no outbound internet access, you must create and use a VPC endpoint for the EKS Auth API: `com.amazonaws.region-code.eks-auth`. For more information about the EKS and EKS Auth VPC endpoints, see [Access Amazon EKS using AWS PrivateLink](vpc-interface-endpoints.md).
+ **IRSA** - Pods configured with [IAM roles for service accounts](iam-roles-for-service-accounts.md) acquire credentials from an AWS Security Token Service (AWS STS) API call. If there is no outbound internet access, you must create and use an AWS STS VPC endpoint in your VPC. Most AWS `v1` SDKs use the global AWS STS endpoint by default (`sts.amazonaws.com`), which doesn’t use the AWS STS VPC endpoint. To use the AWS STS VPC endpoint, you might need to configure your SDK to use the regional AWS STS endpoint (`sts.region-code.amazonaws.com`). For more information, see [Configure the AWS Security Token Service endpoint for a service account](configure-sts-endpoint.md).
+ Your cluster’s VPC subnets must have a VPC interface endpoint for any AWS services that your Pods need access to. For more information, see [Access an AWS service using an interface VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html). Some commonly-used services and endpoints are listed in the following table. For a complete list of endpoints, see [AWS services that integrate with AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.html) in the [AWS PrivateLink Guide](https://docs.aws.amazon.com/vpc/latest/privatelink/).
We recommend that you [enable private DNS names](https://docs.aws.amazon.com/vpc/latest/privatelink/interface-endpoints.html#enable-private-dns-names) for your VPC endpoints, that way workloads can continue using public AWS service endpoints without issues.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/eks/latest/userguide/private-clusters.html)
+ Any self-managed nodes must be deployed to subnets that have the VPC interface endpoints that you require. If you create a managed node group, the VPC interface endpoint security group must allow the CIDR for the subnets, or you must add the created node security group to the VPC interface endpoint security group.
+ **EFS storage** - If your Pods use Amazon EFS volumes, then before deploying the [Store an elastic file system with Amazon EFS](efs-csi.md), the driver’s [kustomization.yaml](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/deploy/kubernetes/overlays/stable/kustomization.yaml) file must be changed to set the container images to use the same AWS Region as the Amazon EKS cluster.
+ Route53 does not support AWS PrivateLink. You cannot manage Route53 DNS records from a private Amazon EKS cluster. This impacts Kubernetes [external-dns](https://github.com/kubernetes-sigs/external-dns).
+ If you use the EKS Optimized AMI, you should enable the `ec2` endpoint in the table above. Alternatively, you can manually set the Node DNS name. The optimized AMI uses EC2 APIs to set the node DNS name automatically.
+ You can use the [AWS Load Balancer Controller](aws-load-balancer-controller.md) to deploy AWS Application Load Balancers (ALB) and Network Load Balancers to your private cluster. When deploying it, you should use [command line flags](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/configurations/#controller-command-line-flags) to set `enable-shield`, `enable-waf`, and `enable-wafv2` to false. [Certificate discovery](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/cert_discovery/#discover-via-ingress-rule-host) with hostnames from Ingress objects isn’t supported. This is because the controller needs to reach AWS Certificate Manager, which doesn’t have a VPC interface endpoint.
The controller supports network load balancers with IP targets, which are required for use with Fargate. For more information, see [Route application and HTTP traffic with Application Load Balancers](alb-ingress.md) and [Create a network load balancer](network-load-balancing.md#network-load-balancer).
+ [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md) is supported. When deploying Cluster Autoscaler Pods, make sure that the command line includes `--aws-use-static-instance-list=true`. For more information, see [Use Static Instance List](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#use-static-instance-list) on GitHub. The worker node VPC must also include the AWS STS VPC endpoint and autoscaling VPC endpoint.
+ Some container software products use API calls that access the AWS Marketplace Metering Service to monitor usage. Private clusters do not allow these calls, so you can’t use these container types in private clusters.
# Scale cluster compute with Karpenter and Cluster Autoscaler
Autoscaling is a function that automatically scales your resources out and in to meet changing demands. This is a major Kubernetes function that would otherwise require extensive human resources to perform manually.
## EKS Auto Mode
Amazon EKS Auto Mode automatically scales cluster compute resources. If a pod can’t fit onto existing nodes, EKS Auto Mode creates a new one. EKS Auto Mode also consolidates workloads and deletes nodes. EKS Auto Mode builds upon Karpenter.
For more information, see:
+ [Automate cluster infrastructure with EKS Auto Mode](automode.md)
+ [Create a Node Pool for EKS Auto Mode](create-node-pool.md)
+ [Deploy a sample inflate workload to an Amazon EKS Auto Mode cluster](automode-workload.md)
## Additional Solutions
Amazon EKS supports two additional autoscaling products:
**Karpenter**
Karpenter is a flexible, high-performance Kubernetes cluster autoscaler that helps improve application availability and cluster efficiency. Karpenter launches right-sized compute resources (for example, Amazon EC2 instances) in response to changing application load in under a minute. Through integrating Kubernetes with AWS, Karpenter can provision just-in-time compute resources that precisely meet the requirements of your workload. Karpenter automatically provisions new compute resources based on the specific requirements of cluster workloads. These include compute, storage, acceleration, and scheduling requirements. Amazon EKS supports clusters using Karpenter, although Karpenter works with any conformant Kubernetes cluster. For more information, see the [Karpenter](https://karpenter.sh/docs/) documentation.
Karpenter is open-source software which AWS customers are responsible for installing, configuring, and managing in their Kubernetes clusters. AWS provides technical support when Karpenter is run unmodified using a compatible version in Amazon EKS clusters. It is essential that customers maintain the availability and security of the Karpenter controller as well as appropriate testing procedures when upgrading it or the Kubernetes cluster in which it’s running, just like any other customer-managed software. There is no AWS Service Level Agreement (SLA) for Karpenter and customers are responsible for ensuring that the EC2 instances launched by Karpenter meet their business requirements.
**Cluster Autoscaler**
The Kubernetes Cluster Autoscaler automatically adjusts the number of nodes in your cluster when pods fail or are rescheduled onto other nodes. The Cluster Autoscaler uses Auto Scaling groups. For more information, see [Cluster Autoscaler on AWS](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md).
# Learn about Amazon Application Recovery Controller (ARC) zonal shift in Amazon EKS
Kubernetes has native features that enable you to make your applications more resilient to events such as the degraded health or impairment of an Availability Zone (AZ). When you run your workloads in an Amazon EKS cluster, you can further improve your application environment’s fault tolerance and application recovery by using [Amazon Application Recovery Controller (ARC) zonal shift](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-shift.html) or [zonal autoshift](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-autoshift.html). ARC zonal shift is designed to be a temporary measure that enables you to move traffic for a resource away from an impaired AZ until the zonal shift expires or you cancel it. You can extend the zonal shift, if necessary.
You can start a zonal shift for an EKS cluster, or you can allow AWS to shift traffic for you by enabling zonal autoshift. This shift updates the flow of east-to-west network traffic in your cluster to only consider network endpoints for Pods running on worker nodes in healthy AZs. Additionally, any ALB or NLB handling ingress traffic for applications in your EKS cluster will automatically route traffic to targets in the healthy AZs. For those customers seeking the highest availability goals, in the case that an AZ becomes impaired, it can be important to be able to steer all traffic away from the impaired AZ until it recovers. For this, you can also [https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-shift.resource-types.html](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-shift.resource-types.html).
## Understanding east-west network traffic flow between Pods
The following diagram illustrates two example workloads, Orders, and Products. The purpose of this example is to show how workloads and Pods in different AZs communicate.
![\[Illustration of network traffic\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-traffic-flow-before-1.png)
![\[Illustration of network traffic\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-traffic-flow-before-2.png)
1. For Orders to communicate with Products, Orders must first resolve the DNS name of the destination service. Orders communicates with CoreDNS to fetch the virtual IP address (Cluster IP) for that service. After Orders resolves the Products service name, it sends traffic to that target IP address.
1. The kube-proxy runs on every node in the cluster and continuously watches [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/) for services. When a service is created, an EndpointSlice is created and managed in the background by the EndpointSlice controller. Each EndpointSlice has a list or table of endpoints that contains a subset of Pod addresses, along with the nodes that they’re running on. The kube-proxy sets up routing rules for each of these Pod endpoints using `iptables` on the nodes. The kube-proxy is also responsible for a basic form of load balancing, redirecting traffic destined to a service’s Cluster IP address to instead be sent to a Pod’s IP address directly. The kube-proxy does this by rewriting the destination IP address on the outgoing connection.
1. The network packets are then sent to the Products Pod in AZ 2 by using the ENIs on the respective nodes, as shown in the earlier diagram.
### Understanding ARC zonal shift in Amazon EKS
If there is an AZ impairment in your environment, you can initiate a zonal shift for your EKS cluster environment. Alternatively, you can allow AWS to manage shifting traffic for you with zonal autoshift. With zonal autoshift, AWS monitors overall AZ health and responds to a potential AZ impairment by automatically shifting traffic away from the impaired AZ in your cluster environment.
After your Amazon EKS cluster has zonal shift enabled with ARC, you can start a zonal shift or enable zonal autoshift by using the ARC Console, the AWS CLI, or the zonal shift and zonal autoshift APIs. During an EKS zonal shift, the following is performed automatically:
+ All the nodes in the impacted AZ are cordoned. This prevents the Kubernetes Scheduler from scheduling new Pods onto nodes in the unhealthy AZ.
+ If you’re using [Managed Node Groups](managed-node-groups.md), [https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#AutoScalingBehavior.InstanceUsage](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#AutoScalingBehavior.InstanceUsage) is suspended, and your Auto Scaling group is updated to ensure that new EKS data plane nodes are only launched in healthy AZs.
+ The nodes in the unhealthy AZ are not terminated, and Pods are not evicted from the nodes. This ensures that when a zonal shift expires or is canceled, your traffic can be safely returned to the AZ for full capacity.
+ The EndpointSlice controller finds all Pod endpoints in the impaired AZ, and removes them from the relevant EndpointSlices. This ensures that only Pod endpoints in healthy AZs are targeted to receive network traffic. When a zonal shift is canceled or expires, the EndpointSlice controller updates the EndpointSlices to include the endpoints in the restored AZ.
The following diagrams provide a high level overview of how EKS zonal shift ensures that only healthy Pod endpoints are targeted in your cluster environment.
![\[Illustration of network traffic\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-traffic-flow-after-1.png)
![\[Illustration of network traffic\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-traffic-flow-after-2.png)
## EKS zonal shift requirements
For zonal shift to work successfully with EKS, you must set up your cluster environment ahead of time to be resilient to an AZ impairment. The following is a list of configuration options that help to ensure resilience.
+ Provision your cluster’s worker nodes across multiple AZs
+ Provision enough compute capacity to accommodate removal of a single AZ
+ Pre-scale your Pods, including CoreDNS, in every AZ
+ Spread multiple Pod replicas across all AZs, to help ensure that when you shift away from a single AZ, you’ll still have sufficient capacity
+ Colocate interdependent or related Pods in the same AZ
+ Test that your cluster environment works as expected without one AZ by manually starting a zonal shift away from an AZ. Alternatively, you can enable zonal autoshift and rely on autoshift practice runs. Testing with manual or practice zonal shifts is not required for zonal shift to work in EKS but it’s strongly recommended.
### Provision your EKS worker nodes across multiple Availability Zones
AWS Regions have multiple, separate locations with physical data centers, known as Availability Zones (AZs). AZs are designed to be physically isolated from one another to avoid simultaneous impact that could affect an entire Region. When you provision an EKS cluster, we recommend that you deploy your worker nodes across multiple AZs in a Region. This helps to make your cluster environment more resilient to the impairment of a single AZ, and allows you to maintain high availability for your applications that run in the other AZs. When you start a zonal shift away from the impacted AZ, your EKS environment’s in-cluster network automatically updates to only use healthy AZs, to help maintaining high availability for your cluster.
Ensuring that you have a multi-AZ setup for your EKS environment enhances the overall reliability of your system. However, multi-AZ environments influence how application data is transferred and processed, which in turn has an impact on your environment’s network charges. Specifically, frequent egress cross-zone traffic (traffic distributed between AZs) can have a major impact on your network-related costs. You can apply different strategies to control the amount of cross-zone traffic between Pods in your EKS cluster and drive down the associated costs. For more information on how to optimize network costs when running highly available EKS environments, see [https://aws.github.io/aws-eks-best-practices/cost_optimization/cost_opt_networking/](https://aws.github.io/aws-eks-best-practices/cost_optimization/cost_opt_networking/).
The following diagram illustrates a highly-available EKS environment with three healthy AZs.
![\[Illustration of network\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-ha-before-failure.png)
The following diagram illustrates how an EKS environment with three AZs is resilient to an AZ impairment and remains highly available because there are two remaining healthy AZs.
![\[Illustration of network\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-ha-after-failure.png)
### Provision enough compute capacity to withstand removal of a single Availability Zone
To optimize resource utilization and costs for your compute infrastructure in the EKS data plane, it’s a best practice to align compute capacity with your workload requirements. However, **if all your worker nodes are at full capacity**, you are reliant on having new worker nodes added to the EKS data plane before new Pods can be scheduled. When you run critical workloads, it is generally a good practice to run with redundant capacity online to handle scenarios such as sudden increases in load and node health issues. If you plan to use zonal shift, you are planning to remove an entire AZ of capacity when there’s an impairment. This means that you must adjust your redundant compute capacity so that it’s sufficient to handle the load even with one of the AZs offline.
When you scale your compute resources, the process of adding new nodes to the EKS data plane takes some time. This can have implications on the real-time performance and availability of your applications, especially in the event of a zonal impairment. Your EKS environment should be able to absorb the load of losing one AZ without resulting in a degraded experience for your end users or clients. This means minimizing or eliminating lag between the time when a new Pod is needed and when it’s actually scheduled on a worker node.
Additionally, when there’s a zonal impairment, you should aim to mitigate the risk of running into a compute capacity constraint that would prevent newly-required nodes from being added to your EKS data plane in the healthy AZs.
To accomplish reduce the risk of these potential negative impacts, we recommend that you over-provision compute capacity in some of the worker nodes in each of the AZs. By doing this, the Kubernetes Scheduler has pre-existing capacity available for new Pod placements, which is especially important when you lose one of the AZs in your environment.
### Run and spread multiple Pod replicas across Availability Zones
Kubernetes allows you to pre-scale your workloads by running multiple instances (Pod replicas) of a single application. Running multiple Pod replicas for an application eliminates single points of failure and increases overall performance by reducing the resource strain on a single replica. However, to have both high availability and better fault tolerance for your applications, we recommend that you run multiple replicas of your application and spread the replicas across different failure domains, also referred to as topology domains. The failure domains in this scenario are the Availability Zones. By using [topology spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/), you can set up your applications to have pre-existing, static stability. Then, when there’s an AZ impairment, your environment will have enough replicas in healthy AZs to immediately handle any spikes or surges in traffic.
The following diagram illustrates an EKS environment that has east-to-west traffic flow when all AZs are healthy.
![\[Illustration of network\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-spread-constraints.png)
The following diagram illustrates an EKS environment that has east-to-west traffic flow where a single AZ has failed and you have started a zonal shift.
![\[Illustration of network\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-spread-constraints-2.png)
The following code snippet is an example of how to set up your workload with multiple replicas in Kubernetes.
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
spec:
replicas: 9
selector:
matchLabels:
app:orders
template:
metadata:
labels:
app: orders
tier: backend
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: "topology.kubernetes.io/zone"
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: orders
```
Most importantly, you should run multiple replicas of your DNS server software (CoreDNS/kube-dns) and apply similar topology spread constraints, if they are not configured by default. This helps to ensure that, if there’s a single AZ impairment, you have enough DNS Pods in healthy AZs to continue handling service discovery requests for other communicating Pods in the cluster. The [CoreDNS EKS add-on](managing-coredns.md) has default settings for the CoreDNS Pods that ensure that, if there are nodes in multiple AZs available, they are spread across your cluster’s Availability Zones. If you like, you can replace these default settings with your own custom configurations.
When you install [CoreDNS with Helm](https://github.com/coredns/helm/tree/master), you can update the `replicaCount` in the [values.yaml file](https://github.com/coredns/helm/blob/master/charts/coredns/values.yaml) to ensure that you have sufficient replicas in each AZ. In addition, to ensure that these replicas are spread across the different AZs in your cluster environment, make sure that you update the `topologySpreadConstraints` property in the same `values.yaml` file. The following code snippet illustrates how you can configure CoreDNS to do this.
**CoreDNS Helm values.yaml**
```
replicaCount: 6
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
k8s-app: kube-dns
```
If there’s an AZ impairment, you can absorb the increased load on the CoreDNS Pods by using an autoscaling system for CoreDNS. The number of DNS instances that you will require depends on the number of workloads that are running in your cluster. CoreDNS is CPU bound, which allows it to scale based on CPU by using the [Horizontal Pod Autoscaler (HPA)](https://aws.github.io/aws-eks-best-practices/reliability/docs/application/#horizontal-pod-autoscaler-hpa). The following is an example that you can modify to suit your needs.
```
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: coredns
namespace: default
spec:
maxReplicas: 20
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: coredns
targetCPUUtilizationPercentage: 50
```
Alternatively, EKS can manage autoscaling of the CoreDNS deployment in the EKS add-on version of CoreDNS. This CoreDNS autoscaler continuously monitors the cluster state, including the number of nodes and CPU cores. Based on that information, the controller dynamically adjusts the number of replicas of the CoreDNS deployment in an EKS cluster.
To enable the [autoscaling configuration in the CoreDNS EKS add-on](coredns-autoscaling.md), use the following configuration setting:
```
{
"autoScaling": {
"enabled": true
}
}
```
You can also use [NodeLocal DNS](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/) or the [cluster proportional autoscaler](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler) to scale CoreDNS. For more information, see [Scaling CoreDNS horizontally](https://aws.github.io/aws-eks-best-practices/scalability/docs/cluster-services/#scale-coredns-horizontally).
### Colocate interdependent Pods in the same Availability Zone
Typically, applications have distinct workloads that need to communicate with each other to successfully complete an end-to-end process. If these distinct applications are spread across different AZs and are not colocated in the same AZ, then a single AZ impairment can impact the end-to-end process. For example, if **Application A** has multiple replicas in AZ 1 and AZ 2, but **Application B** has all its replicas in AZ 3, then the loss of AZ 3 will affect end-to-end processes between the two workloads, **Application A** and **Application B**. If you combine topology spread constraints with pod affinity, you can enhance your application’s resiliency by spreading Pods across all AZs. In addition, this configures a relationship between certain Pods to ensure that they’re colocated.
With [pod affinity rules](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/), you can define relationships between workloads to influence the behavior of the Kubernetes Scheduler so that it colocates Pods on the same worker node or in the same AZ. You can also configure how strict the scheduling constraints should be.
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: products
namespace: ecommerce
labels:
app.kubernetes.io/version: "0.1.6"
spec:
serviceAccountName: graphql-service-account
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- orders
topologyKey: "kubernetes.io/hostname"
```
The following diagram shows several pods that have been colocated on the same node by using pod affinity rules.
![\[Illustration of network\]](http://docs.aws.amazon.com/eks/latest/userguide/images/zs-pod-affinity-rule.png)
### Test that your cluster environment can handle the loss of an AZ
After you complete the requirements described in the previous sections, the next step is to test that you have sufficient compute and workload capacity to handle the loss of an AZ. You can do this by manually starting a zonal shift in EKS. Alternatively, you can enable zonal autoshift and configure practice runs, which also test that your applications function as expected with one less AZ in your cluster environment.
## Frequently asked questions
**Why should I use this feature?**
By using ARC zonal shift or zonal autoshift in your EKS cluster, you can better maintain Kubernetes application availability by automating the quick recovery process of shifting in-cluster network traffic away from an impaired AZ. With ARC, you can avoid long, complicated steps that can lead to an extended recovery period during impaired AZ events.
**How does this feature work with other AWS services?**
EKS integrates with ARC, which provides the primary interface for you to accomplish recovery operations in AWS. To ensure that in-cluster traffic is appropriately routed away from an impaired AZ, EKS makes modifications to the list of network endpoints for Pods running in the Kubernetes data plane. If you’re using Elastic Load Balancing to route external traffic into the cluster, you can register your load balancers with ARC and start a zonal shift on them to prevent traffic from flowing into the degraded AZ. Zonal shift also works with Amazon EC2 Auto Scaling groups that are created by EKS managed node groups. To prevent an impaired AZ from being used for new Kubernetes Pods or node launches, EKS removes the impaired AZ from the Auto Scaling groups.
**How is this feature different from default Kubernetes protections?**
This feature works in tandem with several Kubernetes built-in protections that help customer applications' resiliency. You can configure Pod readiness and liveness probes that decide when a Pod should take traffic. When these probes fail, Kubernetes removes these Pods as targets for a service, and traffic is no longer sent to the Pod. While this is useful, it’s not simple for customers to configure these health checks so that they are guaranteed to fail when an AZ is degraded. The ARC zonal shift feature provides an additional safety net that helps you to isolate a degraded AZ entirely when Kubernetes' native protections were not enough. Zonal shift also gives you an easy way to test the operational readiness and resilience of your architecture.
**Can AWS start a zonal shift on my behalf?**
Yes, if you want a fully automated way of using ARC zonal shift, you can enable ARC zonal autoshift. With zonal autoshift, you can rely on AWS to monitor the health of the AZs for your EKS cluster, and to automatically start a zonal shift when an AZ impairment is detected.
**What happens if I use this feature and my worker nodes and workloads are not pre-scaled?**
If you are not pre-scaled and rely on provisioning additional nodes or Pods during a zonal shift, you risk a delayed recovery. The process of adding new nodes to the Kubernetes data plane takes some time, which can impact the real-time performance and availability of your applications, especially when there’s a zonal impairment. Additionally, in the event of a zonal impairment, you may encounter a potential compute capacity constraint that could prevent newly required nodes from being added to the healthy AZs.
If your workloads are not pre-scaled and spread across all AZs in your cluster, a zonal impairment might impact the availability of an application that is only running on worker nodes in an impacted AZ. To mitigate the risk of a complete availability outage for your application, EKS has a fail safe for traffic to be sent to Pod endpoints in an impaired zone if that workload has all of its endpoints in the unhealthy AZ. However, we strongly recommend that you pre-scale and spread your applications across all AZs to maintain availability in the event of a zonal issue.
**How does this work if I’m running a stateful application?**
If you are running a stateful application, you must assess its fault tolerance, based on your use case and architecture. If you have an active/standby architecture or pattern, there might be instances where the active is in an impaired AZ. At the application level, if the standby is not activated, you might run into issues with your application. You might also run into issues when new Kubernetes Pods are launched in healthy AZs, since they won’t be able to attach to the persistent volumes bounded to the impaired AZ.
**Does this feature work with Karpenter?**
Karpenter support is currently not available with ARC zonal shift and zonal autoshift in EKS. If an AZ is impaired, you can adjust the relevant Karpenter NodePool configuration by removing the unhealthy AZ so that new worker nodes are only launched in the other AZs.
**Does this feature work with EKS Fargate?**
This feature does not work with EKS Fargate. By default, when EKS Fargate recognizes a zonal health event, Pods will prefer to run in the other AZs.
**Will the EKS managed Kubernetes control plane be impacted?**
No, by default Amazon EKS runs and scales the Kubernetes control plane across multiple AZs to ensure high availability. ARC zonal shift and zonal autoshift only act on the Kubernetes data plane.
**Are there any costs associated with this new feature?**
You can use ARC zonal shift and zonal autoshift in your EKS cluster at no additional charge. However, you will continue to pay for provisioned instances and we strongly recommended that you pre-scale your Kubernetes data plane before using this feature. You should consider a balance between cost and application availability.
## Additional resources
+ [How a zonal shift works](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-shift.how-it-works.html)
+ [Best practices for zonal shifts in ARC](https://docs.aws.amazon.com/r53recovery/latest/dg/route53-arc-best-practices.zonal-shifts.html#zonalshift.route53-arc-best-practices.zonal-shifts)
+ [Resources and scenarios supported for zonal shift and zonal autoshift](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-shift.resource-types.html)
+ [Operating resilient workloads on Amazon EKS](https://aws.amazon.com/blogs/containers/operating-resilient-workloads-on-amazon-eks)
+ [Eliminate Kubernetes node scaling lag with pod priority and over-provisioning](https://aws.amazon.com/blogs/containers/eliminate-kubernetes-node-scaling-lag-with-pod-priority-and-over-provisioning)
+ [Scale CoreDNS Pods for high DNS traffic](coredns-autoscaling.md)
# Enable EKS zonal shift to avoid impaired Availability Zones
Amazon Application Recovery Controller (ARC) helps you manage and coordinate recovery for your applications across Availability Zones (AZs) and works with many services, including Amazon EKS. With EKS support for ARC zonal shift, you can shift in-cluster network traffic away from an impaired AZ. You can also authorize AWS to monitor the health of your AZs and temporarily shift network traffic away from an unhealthy AZ on your behalf.
**How to use EKS zonal shift:**
1. Enable your EKS cluster with Amazon Application Recovery Controller (ARC). This is done at the cluster level using the Amazon EKS Console, the AWS CLI, CloudFormation, or eksctl.
1. Once enabled, you can manage zonal shifts or zonal autoshifts using the ARC Console, the AWS CLI, or the zonal shift and zonal autoshift APIs.
Note that after you register an EKS cluster with ARC, you still need to configure ARC. For example, you can use the ARC console to configure zonal autoshift.
For more detailed information about how EKS zonal shift works, and how to design your workloads to handle impaired availability zones, see [Learn about Amazon Application Recovery Controller (ARC) zonal shift in Amazon EKS](zone-shift.md).
## Considerations
+ EKS Auto Mode does not support Amazon Application Recovery Controller, zonal shift, and zonal autoshift.
+ We recommend waiting at least 60 seconds between zonal shift operations to ensure proper processing of each request.
When attempting to perform zonal shifts in quick succession (within 60 seconds of each other), the Amazon EKS service may not properly process all shift requests. This is due to the current polling mechanism that updates the cluster’s zonal state. If you need to perform multiple zonal shifts, ensure there is adequate time between operations for the system to process each change.
## What is Amazon Application Recovery Controller?
Amazon Application Recovery Controller (ARC) helps you prepare for and accomplish faster recovery for applications running on AWS. Zonal shift enables you to quickly recover from Availability Zone (AZ) impairments, by temporarily moving traffic for a supported resource away from an AZ, to healthy AZs in the AWS Region.
[Learn more about Amazon Application Recovery Controller (ARC)](https://docs.aws.amazon.com/r53recovery/latest/dg/what-is-route53-recovery.html)
## What is zonal shift?
Zonal shift is a capability in ARC that allows you to move traffic for a resource like an EKS cluster or an Elastic Load Balancer away from an Availability Zone in an AWS Region to quickly mitigate an issue and quickly recover your application. You might choose to shift traffic, for example, because a bad deployment is causing latency issues, or because the Availability Zone is impaired. A zonal shift requires no advance configuration steps.
[Learn more about ARC zonal shift](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-shift.how-it-works.html)
## What is zonal autoshift?
Zonal autoshift is a capability in ARC that you can enable to authorize AWS to shift traffic away from an AZ for supported resources, on your behalf, to healthy AZs in the AWS Region. AWS starts an autoshift when internal telemetry indicates that there is an impairment in one AZ in a Region that could potentially impact customers. The internal telemetry incorporates metrics from multiple sources, including the AWS network, and the Amazon EC2 and Elastic Load Balancing services.
AWS ends autoshifts when indicators show that there is no longer an issue or potential issue.
[Learn more about ARC zonal autoshift](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-autoshift.how-it-works.html)
## What does EKS do during an autoshift?
EKS updates networking configurations to avoid directing traffic to impaired AZs. Additionally, if you are using Managed Node Groups, EKS will only launch new nodes in the healthy AZs during a zonal shift. When the shift expires or gets cancelled, the networking configurations will be restored to include the AZ that was previously detected as unhealthy.
[Learn more about EKS zonal shift](zone-shift.md).
## Register EKS cluster with Amazon Application Recovery Controller (ARC) (AWS console)
1. Find the name and region of the EKS cluster you want to register with ARC.
1. Navigate to the [EKS console](https://console.aws.amazon.com/eks) in that region, and select your cluster.
1. On the **Cluster info** page, select the **Overview** tab.
1. Under the **Zonal shift** heading, select the **Manage** button.
1. Select **enable** or **disable** for *EKS zonal shift*.
Now your EKS cluster is registered with ARC.
If you want AWS to detect and avoid impaired availability zones, you need to configure ARC zonal autoshift. For example, you can do this in the ARC console.
## Next Steps
+ Learn how to [enable zonal autoshift](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-autoshift.start-cancel.html).
+ Learn how to manually [start a zonal shift](https://docs.aws.amazon.com/r53recovery/latest/dg/arc-zonal-shift.start-cancel.html).