Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Prevent Pods from being scheduled on accelerated nodes

Overview

Nodes with specialized processors, such as GPUs, can be more expensive to run than nodes on standard machines. To protect these nodes from workloads that don’t require special hardware, you can use Kubernetes taints. Taints mark nodes to repel Pods that don’t have matching tolerations, ensuring only compatible workloads are scheduled. For more information, see Taints and Tolerations in the Kubernetes documentation.

A taint is the complement of a node label. A label lets a workload opt in to a kind of node, but it does nothing to keep other workloads off. Only a taint makes exclusion the default, so that a Pod has to declare a toleration to land on the node. Use both together: taint the accelerated nodes so nothing lands there by accident, and label them so the workloads that belong there can find them. For the label side, see Target AI/ML workloads to specific nodes with labels.

A convention for the taint key

Any key works, but a consistent one is worth more than a descriptive one, because every workload and every add-on DaemonSet that needs an accelerated node has to spell the same key. The convention used throughout the Amazon EKS AI/ML guidance is to taint on the resource the node provides, with no value:

taints:
  - key: nvidia.com/gpu
    effect: NoSchedule

Workloads then tolerate it with operator: Exists, which matches the key regardless of value:

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

This pairing is what the Terraform and manifest examples in the sample-eks-docs repository on GitHub use for every GPU node pool and GPU workload. Reusing it means the manifests in that repository, and the add-on Helm values that ship with it, work against your cluster without modification. For AWS Trainium and AWS Inferentia nodes, apply the same pattern with the aws.amazon.com/neuron key.

Considerations

Prerequisites

Step 1: Taint the accelerated nodes

Where you declare the taint depends on how you provision nodes. In all three cases the taint is applied as the node joins the cluster, so there is no window in which an untainted accelerated node is schedulable. Choose the tab for the way you provision nodes.

EKS Auto Mode

Declare the taint under spec.template.spec.taints on the NodePool. Every node the NodePool provisions carries it.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        workload-class: gpu-inference
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      requirements:
        - key: eks.amazonaws.com/instance-category
          operator: In
          values: ["g"]
        - key: eks.amazonaws.com/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]

Apply the change with kubectl apply. Existing nodes are not retroactively tainted; EKS Auto Mode treats the change as drift and replaces them. For more information about NodePool configuration, see Create a Node Pool for EKS Auto Mode.

Self-managed Karpenter

The field is the same as for EKS Auto Mode, because both use the karpenter.sh/v1 NodePool API. Only the nodeClassRef and the instance requirement label prefix differ.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        workload-class: gpu-inference
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["g"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]

EKS managed node groups

Kubernetes node taints can be applied to new and existing managed node groups using the AWS Management Console or through the Amazon EKS API. The following example uses the AWS CLI. For information about creating a node group with a taint using the AWS Management Console, see Create a managed node group for your cluster.

Note the NO_SCHEDULE spelling. The Amazon EKS API uses uppercase, underscore-separated effects, unlike the Kubernetes NoSchedule form used in the NodePool examples.

aws eks create-nodegroup \
 --cli-input-json '
{
  "clusterName": "my-cluster",
  "nodegroupName": "gpu",
  "subnets": [
     "subnet-1234567890abcdef0",
     "subnet-abcdef01234567890",
     "subnet-021345abcdef67890"
   ],
  "nodeRole": "arn:aws:iam::111122223333:role/AmazonEKSNodeRole",
  "instanceTypes": ["g6e.2xlarge"],
  "amiType": "AL2023_x86_64_NVIDIA",
  "labels": {
     "workload-class": "gpu-inference"
   },
  "taints": [
     {
         "key": "nvidia.com/gpu",
         "effect": "NO_SCHEDULE"
     }
   ]
}'

To add, remove, or replace taints on an existing node group, use the aws eks update-nodegroup-config command with addOrUpdateTaints or removeTaints. Unlike the NodePool paths, this updates the taint on nodes that are already running, without replacing them.

aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name gpu \
  --taints 'addOrUpdateTaints=[{key=nvidia.com/gpu,effect=NO_SCHEDULE}]'

Confirm the taint reached the nodes:

kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key'

Step 2: Tolerate the taint in your workload

A tainted node repels every Pod that does not tolerate it, including your own GPU workloads. Add the toleration to the Pod spec, and pair it with a node selector so the Pod is both allowed onto the accelerated nodes and directed to them.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inference
  template:
    metadata:
      labels:
        app: inference
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        workload-class: gpu-inference
      containers:
        - name: app
          image: public.ecr.aws/docker/library/busybox:latest
          command: ["sh", "-c", "sleep 3600"]
          resources:
            limits:
              nvidia.com/gpu: "1"

Step 3: Give add-on DaemonSets a matching toleration

Tainting accelerated nodes also repels the DaemonSets that have to run on those nodes, such as device plugins, EFA plugins, and metrics exporters. A device plugin that cannot schedule onto a GPU node never advertises the GPU resource, so the node reports no allocatable GPUs and your workloads stay Pending. This failure is easy to misread, because the node looks healthy.

EKS Auto Mode manages its own device plugins and tolerates its NodePool taints without configuration. For any component you install yourself, add the toleration to the chart values or the DaemonSet spec:

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

If workloads are Pending on a node that looks healthy, check whether the node advertises the resource at all:

kubectl get node <node-name> -o jsonpath='{.status.allocatable}'

Related topics

Taints repel workloads from nodes. To steer workloads toward nodes, use node labels: