Migrate applications to a new node group

Retrieve the name of your existing node groups, replacing my-cluster with your cluster name.

eksctl get nodegroups --cluster=my-cluster

An example output is as follows.

CLUSTER      NODEGROUP          CREATED               MIN SIZE      MAX SIZE     DESIRED CAPACITY     INSTANCE TYPE     IMAGE ID
default      standard-nodes   2019-05-01T22:26:58Z  1             4            3                    t3.medium         ami-05a71d034119ffc12

Launch a new node group with eksctl with the following command. In the command, replace every example value with your own values. The version number can’t be later than the Kubernetes version for your control plane. Also, it can’t be more than two minor versions earlier than the Kubernetes version for your control plane. We recommend that you use the same version as your control plane.

We recommend blocking Pod access to IMDS if the following conditions are true:

eksctl create nodegroup \
  --cluster my-cluster \
  --version 1.30 \
  --name standard-nodes-new \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4 \
  --managed=false

When the previous command completes, verify that all of your nodes have reached the Ready state with the following command:

kubectl get nodes

Delete the original node group with the following command. In the command, replace every example value with your cluster and node group names:

eksctl delete nodegroup --cluster my-cluster --name standard-nodes-old

Launch a new node group by following the steps that are outlined in Create self-managed Amazon Linux nodes.

When your stack has finished creating, select it in the console and choose Outputs.

Record the NodeInstanceRole for the node group that was created. You need this to add the new Amazon EKS nodes to your cluster.

Update the security groups for both node groups so that they can communicate with each other. For more information, see View Amazon EKS security group requirements for clusters.

Edit the aws-auth configmap to map the new node instance role in RBAC.

kubectl edit configmap -n kube-system aws-auth

Add a new mapRoles entry for the new node group. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.

apiVersion: v1
data:
  mapRoles: |
    - rolearn: ARN of instance role (not instance profile)
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes>
    - rolearn: arn:aws:iam::111122223333:role/nodes-1-16-NodeInstanceRole-U11V27W93CX5
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

Replace the ARN of instance role (not instance profile) snippet with the NodeInstanceRole value that you recorded in a previous step. Then, save and close the file to apply the updated configmap.

Watch the status of your nodes and wait for your new nodes to join your cluster and reach the Ready status.

kubectl get nodes --watch

(Optional) If you’re using the Kubernetes Cluster Autoscaler, scale the deployment down to zero (0) replicas to avoid conflicting scaling actions.

kubectl scale deployments/cluster-autoscaler --replicas=0 -n kube-system

Use the following command to taint each of the nodes that you want to remove with NoSchedule. This is so that new Pods aren’t scheduled or rescheduled on the nodes that you’re replacing. For more information, see Taints and Tolerations in the Kubernetes documentation.

kubectl taint nodes node_name key=value:NoSchedule

If you’re upgrading your nodes to a new Kubernetes version, you can identify and taint all of the nodes of a particular Kubernetes version (in this case, 1.28) with the following code snippet. The version number can’t be later than the Kubernetes version of your control plane. It also can’t be more than two minor versions earlier than the Kubernetes version of your control plane. We recommend that you use the same version as your control plane.

K8S_VERSION=1.28
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
    echo "Tainting $node"
    kubectl taint nodes $node key=value:NoSchedule
done

Determine your cluster’s DNS provider.

kubectl get deployments -l k8s-app=kube-dns -n kube-system

An example output is as follows. This cluster is using CoreDNS for DNS resolution, but your cluster can return kube-dns instead):

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
coredns   1         1         1            1           31m

If your current deployment is running fewer than two replicas, scale out the deployment to two replicas. Replace coredns with kubedns if your previous command output returned that instead.

kubectl scale deployments/coredns --replicas=2 -n kube-system

Drain each of the nodes that you want to remove from your cluster with the following command:

kubectl drain node_name --ignore-daemonsets --delete-local-data

If you’re upgrading your nodes to a new Kubernetes version, identify and drain all of the nodes of a particular Kubernetes version (in this case, 1.28) with the following code snippet.

K8S_VERSION=1.28
nodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")
for node in ${nodes[@]}
do
    echo "Draining $node"
    kubectl drain $node --ignore-daemonsets --delete-local-data
done

After your old nodes finished draining, revoke the security group inbound rules you authorized earlier. Then, delete the AWS CloudFormation stack to terminate the instances.

Edit the aws-auth configmap to remove the old node instance role from RBAC.

kubectl edit configmap -n kube-system aws-auth

Delete the mapRoles entry for the old node group. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.

apiVersion: v1
data:
  mapRoles: |
    - rolearn: arn:aws:iam::111122223333:role/nodes-1-16-NodeInstanceRole-W70725MZQFF8
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::111122223333:role/nodes-1-15-NodeInstanceRole-U11V27W93CX5
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes>

Save and close the file to apply the updated configmap.

(Optional) If you are using the Kubernetes Cluster Autoscaler, scale the deployment back to one replica.

kubectl scale deployments/cluster-autoscaler --replicas=1 -n kube-system

(Optional) Verify that you’re using the latest version of the Amazon VPC CNI plugin for Kubernetes. You might need to update your CNI version to use the latest supported instance types. For more information, see Amazon VPC CNI.

If your cluster is using kube-dns for DNS resolution (see [migrate-determine-dns-step]), scale in the kube-dns deployment to one replica.

kubectl scale deployments/kube-dns --replicas=1 -n kube-system