Maximize network bandwidth on accelerated computing instances with EFA

To maximize bandwidth on the following accelerated instance types, you can use multiple Elastic Fabric Adapter (EFA) interfaces.

For more information about getting started with GPU accelerated instances, see Performance acceleration with GPU instances.

We recommend that you define a single EFA network interface per network card. To configure these interfaces at launch, we recommend the following settings:

If you are using the Amazon EC2 console, in the Launch Instance Wizard, choose Edit in the Network settings section. Expand Advanced network configuration and choose Add network interface to add the required number of network interfaces. For each network interface, for EFA, select Enable. For all network interfaces, except the primary network interface, for Device index, specify 1. Configure the remaining settings as needed.

If you are using the AWS CLI, use the run-instances command, for --network-interfaces, specify the required number of network interfaces. For each network interface, for InterfaceType, specify efa. For the primary network interface, for NetworkCardIndex and DeviceIndex specify 0. For the remaining network interfaces, for NetworkCardIndex specify a unique value from 1 to 31, and for DeviceIndex specify 1.

The following example command snippet shows a request with 32 EFA network interfaces.

$ aws --region $REGION ec2 run-instances \
 --instance-type p5.48xlarge \
 --count 1 \
 --key-name key_pair_name \
 --image-id ami_id \
 --network-interfaces "NetworkCardIndex=0,DeviceIndex=0,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=1,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=2,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=3,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=4,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=5,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=6,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=7,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=8,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=9,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=10,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=11,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=12,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=13,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=14,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=15,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=16,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=17,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=18,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=19,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=20,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=21,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=22,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=23,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=24,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=25,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=26,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=27,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=28,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=29,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=30,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa" \
                      "NetworkCardIndex=31,DeviceIndex=1,Groups=security_group_id,SubnetId=subnet_id,InterfaceType=efa"
...

If you are using a launch template, specify the required number of network interfaces in the launch template. For each network interface, for InterfaceType, specify efa. For the primary network interface, for NetworkCardIndex and DeviceIndex specify 0. For the remaining network interfaces, for NetworkCardIndex specify a unique value from 1 to 31, and for DeviceIndex specify 1. The following snippet shows an example with 3 network interfaces out of the possible 32 network interfaces.

"NetworkInterfaces":[
{
  "NetworkCardIndex":0,
  "DeviceIndex":0,
  "InterfaceType": "efa",
  "AssociatePublicIpAddress":false,
  "Groups":[
    "security_group_id"
  ],
  "DeleteOnTermination":true
},
{
  "NetworkCardIndex": 1,
  "DeviceIndex": 1,
  "InterfaceType": "efa",
  "AssociatePublicIpAddress":false,
  "Groups":[
    "security_group_id"
  ],
  "DeleteOnTermination":true
},
{
  "NetworkCardIndex": 2,
  "DeviceIndex": 1,
  "InterfaceType": "efa",
  "AssociatePublicIpAddress":false,
  "Groups":[
    "security_group_id"
  ],
  "DeleteOnTermination":true
}
...

When launching a P5 or P5e instance with more than one network interface, you cannot auto-assign public IP addresses. However, you can attach an Elastic IP address to the primary network interface (NetworkCardIndex=0, DeviceIndex=0) after launch for internet connectivity. Both Ubuntu 20.04 and later and Amazon Linux 2 and later are configured to use the primary network interface for internet traffic when the instance is launched as recommended on this page.

To achieve the highest network performance on G6e instances, you can use IMDS to map out attached network interfaces and optimize by using networking instances on disjoint NetworkCardIndexes.

The following example script pulls together details from the attached NetworkCardIndexes.

$ TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-
 aws-ec2-metadata-token-ttl-seconds: 21600")
 count=$(ls -l /sys/class/net/eth* | wc -l)

 for ((i = 0 ; i < ${count} ; i++)); do
  mac=$(cat /sys/class/net/eth$i/address)

  network_card=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" 
"http://169.254.169.254/latest/meta-data/network/interfaces/macs/$mac/
network-card")

   device_number=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" 
"http://169.254.169.254/latest/meta-data/network/interfaces/macs/$mac/
device-number")

   device_bdf=$(realpath /sys/class/net/eth${i}/device)

   echo "eth${i} ${network_card} ${device_number} ${device_bdf}"
done