Troubleshoot problems with Amazon EKS clusters and nodes

This chapter covers some common errors that you may see while using Amazon EKS and how to work around them. If you need to troubleshoot specific Amazon EKS areas, see the separate Troubleshooting IAM, Troubleshoot Amazon EKS Connector issues, and Troubleshooting for ADOT using EKS Add-Ons topics.

For other troubleshooting information, see Knowledge Center content about Amazon Elastic Kubernetes Service on AWS re:Post.

Insufficient capacity

If you receive the following error while attempting to create an Amazon EKS cluster, then one of the Availability Zones you specified doesn’t have sufficient capacity to support a cluster.

Cannot create cluster 'example-cluster' because region-1d, the targeted Availability Zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these Availability Zones: region-1a, region-1b, region-1c

Retry creating your cluster with subnets in your cluster VPC that are hosted in the Availability Zones returned by this error message.

There are Availability Zones that a cluster can’t reside in. Compare the Availability Zones that your subnets are in with the list of Availability Zones in the Subnet requirements and considerationsSubnet requirements and considerations.

Nodes fail to join cluster

There are a few common reasons that prevent nodes from joining the cluster:

To identify and troubleshoot common causes that prevent worker nodes from joining a cluster, you can use the AWSSupport-TroubleshootEKSWorkerNode runbook. For more information, see AWSSupport-TroubleshootEKSWorkerNode in the AWS Systems Manager Automation runbook reference.

Unauthorized or access denied (kubectl)

If you receive one of the following errors while running kubectl commands, then you don’t have kubectl configured properly for Amazon EKS or the credentials for the IAM principal (role or user) that you’re using don’t map to a Kubernetes username that has sufficient permissions to Kubernetes objects on your Amazon EKS cluster.

This could be due to one of the following reasons:

If you install and configure the AWS CLI, you can configure the IAM credentials that you use. For more information, see Configuring the AWS CLI in the AWS Command Line Interface User Guide. You can also configure kubectl to use an IAM role, if you assume an IAM role to access Kubernetes objects on your cluster. For more information, see Connect kubectl to an EKS cluster by creating a kubeconfig file.

hostname doesn’t match

Your system’s Python version must be 2.7.9 or later. Otherwise, you receive hostname doesn’t match errors with AWS CLI calls to Amazon EKS. For more information, see What are "hostname doesn’t match" errors? in the Python Requests Frequently Asked Questions.

getsockopt: no route to host

Docker runs in the 172.17.0.0/16 CIDR range in Amazon EKS clusters. We recommend that your cluster’s VPC subnets do not overlap this range. Otherwise, you will receive the following error:

Error: : error upgrading connection: error dialing backend: dial tcp 172.17.<nn>.<nn>:10250: getsockopt: no route to host

Instances failed to join the Kubernetes cluster

If you receive the error Instances failed to join the Kubernetes cluster in the AWS Management Console, ensure that either the cluster’s private endpoint access is enabled, or that you have correctly configured CIDR blocks for public endpoint access. For more information, see Control network access to cluster API server endpoint.

Managed node group error codes

If your managed node group encounters a hardware health issue, Amazon EKS returns an error code to help you to diagnose the issue. These health checks don’t detect software issues because they are based on Amazon EC2 health checks. The following list describes the error codes.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks:node-manager
rules:
- apiGroups:
  - ''
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
  - delete
- apiGroups:
  - ''
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
  - patch
- apiGroups:
  - ''
  resources:
  - pods/eviction
  verbs:
  - create

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks:node-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eks:node-manager
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: eks:node-manager

kubectl describe clusterrole eks:node-manager

kubectl describe clusterrolebinding eks:node-manager

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks:node-manager
rules:
- apiGroups:
  - ''
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
  - delete
- apiGroups:
  - ''
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
  - patch
- apiGroups:
  - ''
  resources:
  - pods/eviction
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks:node-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eks:node-manager
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: eks:node-manager

kubectl apply -f eks-node-manager-role.yaml

Not authorized for images

One potential cause of a Not authorized for images error message is using a private Amazon EKS Windows AMI to launch Windows managed node groups. After releasing new Windows AMIs, AWS makes AMIs that are older than 4 months private, which makes them no longer accessible. If your managed node group is using a private Windows AMI, consider Update a managed node group for your clusterupdating your Windows managed node group. While we can’t guarantee that we can provide access to AMIs that have been made private, you can request access by filing a ticket with AWS Support. For more information, see Patches in the Amazon EC2 User Guide.

Node is in NotReady state

If your node enters a NotReady status, this likely indicates that the node is unhealthy and unavailable to schedule new Pods. This can occur for various reasons, such as the node lacking sufficient resources for CPU, memory, or available disk space.

For Amazon EKS optimized Windows AMIs, there’s no reservation for compute resources specified by default in the kubelet configuration. To help prevent resource issues, you can reserve compute resources for system processes by providing the kubelet with configuration values for kube-reserved and/or system-reserved. You do this using the -KubeletExtraArgs command-line parameter in the bootstrap script. For more information, see Reserve Compute Resources for System Daemons in the Kubernetes documentation and Bootstrap script configuration parameters in this user guide.

CNI log collection tool

The Amazon VPC CNI plugin for Kubernetes has its own troubleshooting script that is available on nodes at /opt/cni/bin/aws-cni-support.sh. You can use the script to collect diagnostic logs for support cases and general troubleshooting.

Use the following command to run the script on your node:

sudo bash /opt/cni/bin/aws-cni-support.sh

curl -O https://raw.githubusercontent.com/awslabs/amazon-eks-ami/master/log-collector-script/linux/eks-log-collector.sh
sudo bash eks-log-collector.sh

The script collects the following diagnostic information. The CNI version that you have deployed can be earlier than the script version.

      This is version 0.6.1. New versions can be found at https://github.com/awslabs/amazon-eks-ami

Trying to collect common operating system logs...
Trying to collect kernel logs...
Trying to collect mount points and volume information...
Trying to collect SELinux status...
Trying to collect iptables information...
Trying to collect installed packages...
Trying to collect active system services...
Trying to collect Docker daemon information...
Trying to collect kubelet information...
Trying to collect L-IPAMD information...
Trying to collect sysctls information...
Trying to collect networking information...
Trying to collect CNI configuration information...
Trying to collect running Docker containers and gather container data...
Trying to collect Docker daemon logs...
Trying to archive gathered information...

	Done... your bundled logs are located in /var/log/eks_i-0717c9d54b6cfaa19_2020-03-24_0103-UTC_0.6.1.tar.gz

The diagnostic information is collected and stored at:

/var/log/eks_i-0717c9d54b6cfaa19_2020-03-24_0103-UTC_0.6.1.tar.gz

Container runtime network not ready

You may receive a Container runtime network not ready error and authorization errors similar to the following:

4191 kubelet.go:2130] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
4191 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Unauthorized
4191 kubelet_node_status.go:106] Unable to register node "ip-10-40-175-122.ec2.internal" with API server: Unauthorized
4191 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: Failed to list *v1.Service: Unauthorized

This can happen due to one of the following reasons:

TLS handshake timeout

When a node is unable to establish a connection to the public API server endpoint, you may see an error similar to the following error.

server.go:233] failed to run Kubelet: could not init cloud provider "aws": error finding instance i-1111f2222f333e44c: "error listing AWS instances: \"RequestError: send request failed\\ncaused by: Post  net/http: TLS handshake timeout\""

The kubelet process will continually respawn and test the API server endpoint. The error can also occur temporarily during any procedure that performs a rolling update of the cluster in the control plane, such as a configuration change or version update.

To resolve the issue, check the route table and security groups to ensure that traffic from the nodes can reach the public endpoint.

InvalidClientTokenId

If you’re using IAM roles for service accounts for a Pod or DaemonSet deployed to a cluster in a China AWS Region, and haven’t set the AWS_DEFAULT_REGION environment variable in the spec, the Pod or DaemonSet may receive the following error:

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid

To resolve the issue, you need to add the AWS_DEFAULT_REGION environment variable to your Pod or DaemonSet spec, as shown in the following example Pod spec.

apiVersion: v1
kind: Pod
metadata:
  name: envar-demo
  labels:
    purpose: demonstrate-envars
spec:
  containers:
  - name: envar-demo-container
    image: gcr.io/google-samples/node-hello:1.0
    env:
    - name: AWS_DEFAULT_REGION
      value: "region-code"

Node groups must match Kubernetes version before upgrading control plane

Before you upgrade a control plane to a new Kubernetes version, the minor version of the managed and Fargate nodes in your cluster must be the same as the version of your control plane’s current version. The Amazon EKS update-cluster-version API rejects requests until you upgrade all Amazon EKS managed nodes to the current cluster version. Amazon EKS provides APIs to upgrade managed nodes. For information on upgrading a managed node group’s Kubernetes version, see Update a managed node group for your cluster. To upgrade the version of a Fargate node, delete the pod that’s represented by the node and redeploy the pod after you upgrade your control plane. For more information, see Update existing cluster to new Kubernetes version.

When launching many nodes, there are Too Many Requests errors

If you launch many nodes simultaneously, you may see an error message in the Amazon EC2 user data execution logs that says Too Many Requests. This can occur because the control plane is being overloaded with describeCluster calls. The overloading results in throttling, nodes failing to run the bootstrap script, and nodes failing to join the cluster altogether.

Make sure that --apiserver-endpoint, --b64-cluster-ca, and --dns-cluster-ip arguments are being passed to the node’s bootstrap script. When including these arguments, there’s no need for the bootstrap script to make a describeCluster call, which helps prevent the control plane from being overloaded. For more information, see Provide user data to pass arguments to the bootstrap.sh file included with an Amazon EKS optimized Linux/Bottlerocket AMI.

HTTP 401 unauthorized error response on Kubernetes API server requests

You see these errors if a Pod’s service account token has expired on a cluster.

Your Amazon EKS cluster’s Kubernetes API server rejects requests with tokens older than 90 days. In previous Kubernetes versions, tokens did not have an expiration. This means that clients that rely on these tokens must refresh them within an hour. To prevent the Kubernetes API server from rejecting your request due to an invalid token, the Kubernetes client SDK version used by your workload must be the same, or later than the following versions:

You can identify all existing Pods in your cluster that are using stale tokens. For more information, see Service account tokens.

Amazon EKS platform version is more than two versions behind the current platform version

This can happen when Amazon EKS isn’t able to automatically update your cluster’s View Amazon EKS platform versions for each Kubernetes versionplatform version. Though there are many causes for this, some of the common causes follow. If any of these problems apply to your cluster, it may still function, its platform version just won’t be updated by Amazon EKS.

Problem

The Amazon EKS cluster IAM rolecluster IAM role was deleted – This role was specified when the cluster was created. You can see which role was specified with the following command. Replace my-cluster with the name of your cluster.

aws eks describe-cluster --name my-cluster --query cluster.roleArn --output text | cut -d / -f 2

An example output is as follows.

eksClusterRole

Solution

Create a new Amazon EKS cluster IAM rolecluster IAM role with the same name.

Problem

A subnet specified during cluster creation was deleted – The subnets to use with the cluster were specified during cluster creation. You can see which subnets were specified with the following command. Replace my-cluster with the name of your cluster.

aws eks describe-cluster --name my-cluster --query cluster.resourcesVpcConfig.subnetIds

An example output is as follows.

[
"subnet-EXAMPLE1",
"subnet-EXAMPLE2"
]

Solution

Confirm whether the subnet IDs exist in your account.

vpc_id=$(aws eks describe-cluster --name my-cluster --query cluster.resourcesVpcConfig.vpcId --output text)
aws ec2 describe-subnets --filters "Name=vpc-id,Values=$vpc_id" --query "Subnets[*].SubnetId"

An example output is as follows.

[
"subnet-EXAMPLE3",
"subnet-EXAMPLE4"
]

If the subnet IDs returned in the output don’t match the subnet IDs that were specified when the cluster was created, then if you want Amazon EKS to update the cluster, you need to change the subnets used by the cluster. This is because if you specified more than two subnets when you created your cluster, Amazon EKS randomly selects subnets that you specified to create new elastic network interfaces in. These network interfaces enable the control plane to communicate with your nodes. Amazon EKS won’t update the cluster if the subnet it selects doesn’t exist. You have no control over which of the subnets that you specified at cluster creation that Amazon EKS chooses to create a new network interface in.

When you initiate a Kubernetes version update for your cluster, the update can fail for the same reason.

Problem

A security group specified during cluster creation was deleted – If you specified security groups during cluster creation, you can see their IDs with the following command. Replace my-cluster with the name of your cluster.

aws eks describe-cluster --name my-cluster --query cluster.resourcesVpcConfig.securityGroupIds

An example output is as follows.

[
    "sg-EXAMPLE1"
]

If [] is returned, then no security groups were specified when the cluster was created and a missing security group isn’t the problem. If security groups are returned, then confirm that the security groups exist in your account.

Solution

Confirm whether these security groups exist in your account.

vpc_id=$(aws eks describe-cluster --name my-cluster --query cluster.resourcesVpcConfig.vpcId --output text)
aws ec2 describe-security-groups --filters "Name=vpc-id,Values=$vpc_id" --query "SecurityGroups[*].GroupId"

An example output is as follows.

[
"sg-EXAMPLE2"
]

If the security group IDs returned in the output don’t match the security group IDs that were specified when the cluster was created, then if you want Amazon EKS to update the cluster, you need to change the security groups used by the cluster. Amazon EKS won’t update a cluster if the security group IDs specified at cluster creation don’t exist.

When you initiate a Kubernetes version update for your cluster, the update can fail for the same reason.

Cluster health FAQs and error codes with resolution paths

Amazon EKS detects issues with your EKS clusters and the cluster infrastructure and stores it in the cluster health. You can detect, troubleshoot, and address cluster issues more rapidly with the aid of cluster health information. This enables you to create application environments that are more secure and up-to-date. Additionally, it may be impossible for you to upgrade to newer versions of Kubernetes or for Amazon EKS to install security updates on a degraded cluster as a result of issues with the necessary infrastructure or cluster configuration. Amazon EKS can take 3 hours to detect issues or detect that an issue is resolved.

The health of an Amazon EKS cluster is a shared responsibility between Amazon EKS and its users. You are responsible for the prerequisite infrastructure of IAM roles and Amazon VPC subnets, as well as other necessary infrastructure, that must be provided in advance. Amazon EKS detects changes in the configuration of this infrastructure and the cluster.

To access your health of your cluster in the Amazon EKS console, look for a section called Health Issues in the Overview tab of the Amazon EKS cluster detail page. This data will be also be available by calling the DescribeCluster action in the EKS API, for example from within the AWS Command Line Interface.

The first two columns are what are needed for API response values. The third field of the Health ClusterIssue object is resourceIds, the return of which is dependent on the issue type.