Troubleshoot EKS Auto Mode - Amazon EKS
Node Monitoring AgentGet console output from an EC2 managed instance by using the AWS EC2 CLIGet node logs by using the kubectl CLIView resources associated with EKS Auto Mode in the AWS ConsoleView IAM Errors in your AWS account

Troubleshoot EKS Auto Mode

With EKS Auto Mode, AWS assumes more responsibility for EC2 Instances in your AWS account. EKS assumes responsibility for the container runtime on nodes, the operating system on the nodes, and certain controllers. This includes a block storage controller, a load balancing controller, and a compute controller.

You must use AWS and Kubernetes APIs to troubleshoot nodes. You can:

  • Use a Kubernetes NodeDiagnostic resource to retrieve node logs.

  • Use the AWS EC2 CLI command get-console-output to retrieve console output from nodes.

Note

EKS Auto Mode uses EC2 managed instances. You cannot directly access EC2 managed instances, including by SSH.

If you have a problem with a controller, you should research:

  • If the resources associated with that controller are properly formatted and valid.

  • If the AWS IAM and Kubernetes RBAC resources are properly configured for your cluster. For more information, see Learn about identity and access in EKS Auto Mode.

Node Monitoring Agent

EKS Auto Mode includes the Amazon EKS Node Monitoring Agent. You can use this agent to view troubleshooting and debugging information about nodes. The node monitoring agent publishes Kubernetes events and node conditions.

Learn how to view findings from the Node Monitoring Agent.

Get console output from an EC2 managed instance by using the AWS EC2 CLI

This procedure helps with troubleshooting boot-time or kernel-level issues.

First, you need to determine the EC2 Instance ID of the instance associated with your workload. Second, use the AWS CLI to retrieve the console output.

  1. Confirm you have kubectl installed and connected to your cluster

  2. (Optional) Use the name of a Kubernetes Deployment to list the associated pods.

    kubectl get pods -l app=<deployment-name>

  3. Use the name of the Kubernetes Pod to determine the EC2 instance ID of the associated node.

    kubectl get pod <pod-name> -o wide

  4. Use the EC2 instance ID to retrieve the console output.

    aws ec2 get-console-output --instance-id <instance id> --latest --output text

Get node logs by using the kubectl CLI

For information about getting node logs, see Retrieve Node Logs for an EKS Auto Mode Managed Node using Kubectl and S3.

View resources associated with EKS Auto Mode in the AWS Console

You can use the AWS console to view the status of resources associated with your EKS Auto Mode cluster.

  • EBS Volumes

    • View EKS Auto Mode volumes by searching for the tag key eks:eks-cluster-name

  • Load Balancers

    • View EKS Auto Mode load balancers by searching for the tag key eks:eks-cluster-name

  • EC2 Instances

    • View EKS Auto Mode instances by searching for the tag key eks:eks-cluster-name

View IAM Errors in your AWS account

  1. Navigate to CloudTrail console

  2. Select "Event History" from the left navigation pane

  3. Apply error code filters:

    • AccessDenied

    • UnauthorizedOperation

    • InvalidClientTokenId

Look for errors related to your EKS cluster. Use the error messages to update your EKS access entries, Cluster IAM Role, or Node IAM Role. You may need to attach a new policy these roles with permissions for EKS Auto Mode.