Networking
Tip
Register
Consider Higher Network Bandwidth or Elastic Fabric Adapter For Applications with High Inter-Node Communication
For distributed training workloads on Amazon EKS with high inter-node communication demands, consider selecting instances with higher network bandwidth or Elastic Fabric Adapter (EFA). Insufficient network performance can bottleneck data transfer, slowing down machine learning tasks like distributed multi-GPU training. Note that inference workloads don’t typically have high inter-node communication.
Ensure your container image includes NCCL and the aws-ofi-nccl plugin
Node provisioning considerations for EFA workloads
When provisioning EFA-capable nodes, the instances that need to communicate must be in the same Availability Zone (hard requirement). Additionally, AWS recommends launching all EFA-enabled instances in a cluster placement group to minimize the physical distance between them within that single AZ, which gives you the lowest possible latency. A placement group is not required for EFA to function, but is strongly recommended for optimal performance.
The following considerations apply to any EFA-based distributed training deployment on EKS. Below we refer to Karpenter annotations as an example, but the same considerations can be applied to both Managed and Self-managed Node Groups implementations as well.
-
Pin to an AZ. EFA requires all communicating nodes to be in the same AZ, so for example, nodes participating in the same distributed training job must not be spread across zones. You can enforce this by pinning the Pod to an AZ using a
nodeSelectoror pod affinity ontopology.kubernetes.io/zone. Select the AZ where your target instance type has the best availability, or where your Capacity Block is reserved. Be aware that while same-AZ co-location improves inter-node latency, it also increases the blast radius of an AZ-level failure. For long-running training workloads, a single AZ outage or capacity event can wipe out hours of accumulated training progress — an expensive loss. Factor this into your checkpointing strategy and job duration planning. -
Configure the cluster placement group (recommended). Specify the placement group in the EC2NodeClass. Karpenter provisions into it automatically. This is recommended for optimal latency but not strictly required for EFA to function. For Capacity Blocks for ML, placement is handled automatically via UltraClusters — no manual placement group is needed. Note that, in this case, the AZ is already locked and therefore, additional Pod-level or NodePool-level AZ restrictions are not needed.
-
Prevent disruption of multi-node training jobs. Use PDBs or
karpenter.sh/do-not-disrupt: "true"annotations on training pods. Without this, Karpenter’s consolidation may attempt to replace or move EFA workloads mid-job, disrupting the entire distributed training run. SetconsolidationPolicy: WhenEmptyon the NodePool to prevent consolidation of occupied nodes. Review the interaction between these labels andterminationGracePeriodandexpireAfterhere. -
Set appropriate expiration. Configure
expireAfteron the NodePool to a value longer than your longest training job, or disable it for training NodePools entirely. A node expiring mid-training terminates the job. -
Use the correct EFA device plugin version. The EFA device plugin
exposes vpc.amazonaws.com/efaas a schedulable resource. -
Configure security groups. All EFA instances must be in the same security group with a self-referencing rule allowing ALL traffic to/from itself. Without this, EFA traffic fails silently.
Understand Spot instance risks with EFA co-location
Amazon EC2 Spot Instances offer significant cost savings for training workloads (see this section for general Spot best practices with GPUs). However, EFA requires all communicating nodes to reside in the same Availability Zone, and AWS recommends placing them in a cluster placement group for optimal latency. This co-location introduces correlated interruption risk: instances share underlying physical infrastructure within the same AZ (and even more so within a placement group), so a single capacity reclamation event can affect multiple instances simultaneously — potentially interrupting your entire multi-node training job at once rather than a single node.
This is fundamentally different from Spot usage without EFA constraints, where nodes can be spread across AZs and interruptions are statistically independent. With EFA’s same-AZ requirement, a single capacity event can cascade across your training cluster.
If you are pursuing cost savings for GPU training workloads, ensure you have evaluated all available purchase options before committing to Spot. Reserved Instances, On-Demand Capacity Reservations (ODCRs), Savings Plans, and Capacity Blocks for ML can all provide significant discounts while guaranteeing capacity availability — avoiding the correlated interruption risk inherent to Spot with EFA co-location constraints.
Planning for IP Address Consumption on Large GPU Instances
By default, the Amazon VPC CNI plugin pre-allocates IP addresses to ensure pods can be scheduled quickly, keeping one full spare ENI attached and populated with IPs. On large instances, this can result in dozens of IPs being reserved per node even when only a few pods are running.
This mismatch is common in training and inference workloads where pod density per node is low. At cluster scale, especially during autoscaling events that spin up many GPU nodes with few pods each, this can lead to subnet IP exhaustion even though actual IP utilization is low.
To mitigate this, tune the WARM_IP_TARGET, MINIMUM_IP_TARGET, and WARM_ENI_TARGET variables to match your actual pod density. More info at VPC CNI’s ENI and IP target settings
For a full guide on optimizing IP consumption, see Optimizing IP Address Utilization.
