Get started with EFA and MPI for HPC workloads on Amazon EC2

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

In the navigation pane, choose Security Groups and then choose Create security group.

In the Create security group window, do the following:

1. For Security group name, enter a descriptive name for the security group, such as EFA-enabled security group.

2. (Optional) For Description, enter a brief description of the security group.

3. For VPC, select the VPC into which you intend to launch your EFA-enabled instances.

4. Choose Create security group.

Select the security group that you created, and on the Details tab, copy the Security group ID.

With the security group still selected, choose Actions, Edit inbound rules, and then do the following:

1. Choose Add rule.

2. For Type, choose All traffic.

3. For Source type, choose Custom and paste the security group ID that you copied into the field.

4. Choose Add rule.

5. For Type, choose SSH.

6. For Source type, choose Anywhere-IPv4.

7. Choose Save rules.

With the security group still selected, choose Actions, Edit outbound rules, and then do the following:

1. Choose Add rule.

2. For Type, choose All traffic.

3. For Destination type, choose Custom and paste the security group ID that you copied into the field.

4. Choose Save rules.

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

In the navigation pane, choose Instances, and then choose Launch Instances to open the new launch instance wizard.

(Optional) In the Name and tags section, provide a name for the instance, such as EFA-instance. The name is assigned to the instance as a resource tag (Name=EFA-instance).

In the Application and OS Images section, select an AMI for one of the supported operating systems.

In the Instance type section, select a supported instance type.

In the Key pair section, select the key pair to use for the instance.

In the Network settings section, choose Edit, and then do the following:

1. For Subnet, choose the subnet in which to launch the instance. If you do not select a subnet, you can't enable the instance for EFA.

2. For Firewall (security groups), choose Select existing security group, and then select the security group that you created in the previous step.

3. Expand the Advanced network configuration section.

   For Network interface 1, select Network card index = 0, Device index = 0, and Interface type = EFA with ENA.

   (Optional) If you are using a multi-card instance type, such as p4d.24xlarge or p5.48xlarge, for each additional network interface required, choose Add network interface, for Network card index select the next unused index, and then select Device index = 1 and Interface type = EFA with ENA or EFA-only.

In the Storage section, configure the volumes as needed.

In the Summary panel on the right, choose Launch instance.

Connect to the instance you launched. For more information, see Connect to your Linux instance using SSH.

To ensure that all of your software packages are up to date, perform a quick software update on your instance. This process may take a few minutes.

Reboot the instance and reconnect to it.

Download the EFA software installation files. The software installation files are packaged into a compressed tarball (.tar.gz) file. To download the latest stable version, use the following command.

$ curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.37.0.tar.gz

You can also get the latest version by replacing the version number with latest in the preceding command.

Extract the files from the compressed .tar.gz file and navigate into the extracted directory.

$ tar -xf aws-efa-installer-1.37.0.tar.gz && cd aws-efa-installer

Install the EFA software. Do one of the following depending on your use case.

If the EFA installer prompts you to reboot the instance, do so and then reconnect to the instance. Otherwise, log out of the instance and then log back in to complete the installation.

Add Open MPI 5 to the PATH environment variable.

$ module load openmpi5

Verify that Open MPI 5 is enabled for use.

$ which mpicc

The command should return the Open MPI 5 installation directory - /opt/amazon/openmpi5.

(Optional) To ensure that Open MPI 5 is added to PATH environment variable each time the instance starts, do the following:

To download the Intel MPI installation script, do the following

1. Visit the Intel website.

2. In the Intel MPI Library section of the webpage, choose the link for the Intel MPI Library for Linux Offline installer.

Run the installation script that you downloaded in the previous step.

$ sudo bash installation_script_name.sh

In the installer, choose Accept & install.

Read the Intel Improvement Program, choose the appropriate option, and then choose Begin Installation.

When the installation completes, choose Close.

By default, Intel MPI uses its embedded (internal) Libfabric. You can configure Intel MPI to use the Libfabric that ships with the EFA installer instead. Typically, the EFA installer ships with a later version of Libfabric than Intel MPI. In some cases, the Libfabric that ships with the EFA installer is more performant than that of Intel MPI. To configure Intel MPI to use the Libfabric that ships with the EFA installer, do one of the following depending on your shell.

Add the following source command to your shell script to source the vars.sh script from the installation directory to set up the compiler environment each time the instance starts. Do one of the following depending on your shell.

By default, if EFA is not available due to a misconfiguration, Intel MPI defaults to the TCP/IP network stack, which might result in slower application performance. You can prevent this by setting I_MPI_OFI_PROVIDER to efa. This causes Intel MPI to fail with the following error if EFA is not available:

Abort (XXXXXX) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: OtherMPI error,
MPIR_Init_thread (XXX)........:	
MPID_Init (XXXX)..............:
MPIDI_OFI_mpi_init_hook (XXXX):
open_fabric (XXXX)............:
find_provider (XXXX)..........:
OFI fi_getinfo() failed (ofi_init.c:2684:find_provider:

Do one of the following depending on your shell.

By default, Intel MPI doesn't print debugging information. You can specify different verbosity levels to control the debugging information. Possible values (in order of the amount of detail they provide) are: 0 (default), 1, 2, 3, 4, 5. Level 1 and higher prints the libfabric version and libfabric provider. Use libfabric version to check whether Intel MPI is using the internal Libfabric or the Libfabric that ships with the EFA installer. If it's using the internal Libfabric, the version is suffixed with impi. Use libfabric provider to check with Intel MPI is using EFA or the TCP/IP network. If it's using EFA, the value is efa. If it's using TCP/IP, the value is tcp;ofi_rxm.

To enable debugging information, do one of the following depending on your shell.

By default, Intel MPI uses the operating system’s shared memory (shm) for intra-node communication, and it uses Libfabric (ofi) only for inter-node communication. Generally, this configuration provides the best performance. However, in some cases the Intel MPI shm fabric can cause certain applications to hang indefinitely.

To resolve this issue, you can force Intel MPI to use Libfabric for both intra-node and inter-node communication. To do this, do one of the following depending on your shell.

Log out of the instance and then log back in.

To confirm that MPI was successfully installed, run the following command:

$ which mpicc

To confirm that the EFA software components and Libfabric were successfully installed, run the following command.

$ fi_info -p efa -t FI_EP_RDM

The command should return information about the Libfabric EFA interfaces. The following example shows the command output.

provider: efa
    fabric: EFA-fe80::94:3dff:fe89:1b70
    domain: efa_0-rdm
    version: 2.0
    type: FI_EP_RDM
    protocol: FI_PROTO_EFA

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

In the navigation pane, choose Instances.

Select the temporary instance that you created and choose Actions, Image, Create image.

For Create image, do the following:

1. For Image name, enter a descriptive name for the AMI.

2. (Optional) For Image description, enter a brief description of the purpose of the AMI.

3. Choose Create image.

In the navigation pane, choose AMIs.

Locate the AMI tht you created in the list. Wait for the status to change from pending to available before continuing to the next step.

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

In the navigation pane, choose Instances, and then choose Launch Instances to open the new launch instance wizard.

(Optional) In the Name and tags section, provide a name for the instance, such as EFA-instance. The name is assigned to the instance as a resource tag (Name=EFA-instance).

In the Application and OS Images section, choose My AMIs, and then select the AMI that you created in the previous step.

In the Instance type section, select a supported instance type.

In the Key pair section, select the key pair to use for the instance.

In the Network settings section, choose Edit, and then do the following:

1. For Subnet, choose the subnet in which to launch the instance. If you do not select a subnet, you can't enable the instance for EFA.

2. For Firewall (security groups), choose Select existing security group, and then select the security group that you created in the previous step.

3. Expand the Advanced network configuration section.

   For Network interface 1, select Network card index = 0, Device index = 0, and Interface type = EFA with ENA.

   (Optional) If you are using a multi-card instance type, such as p4d.24xlarge or p5.48xlarge, for each additional network interface required, choose Add network interface, for Network card index select the next unused index, and then select Device index = 1 and Interface type = EFA with ENA or EFA-only.

(Optional) In the Storage section, configure the volumes as needed.

In the Advanced details section, for Placement group name, select the cluster placement group into which to launch the instances. If you need to create a new cluster placement group, choose Create new placement group.

In the Summary panel on the right, for Number of instances, enter the number of EFA-enabled instances that you want to launch, and then choose Launch instance.

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

In the navigation pane, choose Instances.

Select the temporary instance that you created and then choose Actions, Instance state, Terminate (delete) instance.

When prompted for confirmation, choose Terminate (delete).

Select one instance in the cluster as the leader node, and connect to it.

Disable strictHostKeyChecking and enable ForwardAgent on the leader node. Open ~/.ssh/config using your preferred text editor and add the following.

Host *
    ForwardAgent yes
Host *
    StrictHostKeyChecking no

Generate an RSA key pair.

$ ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

The key pair is created in the $HOME/.ssh/ directory.

Change the permissions of the private key on the leader node.

$ chmod 600 ~/.ssh/id_rsa
chmod 600 ~/.ssh/config

Open ~/.ssh/id_rsa.pub using your preferred text editor and copy the key.

For each member node in the cluster, do the following:

1. Connect to the instance.

2. Open ~/.ssh/authorized_keys using your preferred text editor and add the public key that you copied earlier.

To test that the passwordless SSH is functioning as expected, connect to your leader node and run the following command.

$ ssh member_node_private_ip

You should connect to the member node without being prompted for a key or password.