**Help improve this page** To contribute to this user guide, choose the **Edit this page on GitHub** link that is located in the right pane of every page. # Use high-performance app storage with Amazon FSx for Lustre The [Amazon FSx for Lustre Container Storage Interface (CSI) driver](https://github.com/kubernetes-sigs/aws-fsx-csi-driver) provides a CSI interface that allows Amazon EKS clusters to manage the lifecycle of Amazon FSx for Lustre file systems. For more information, see the [Amazon FSx for Lustre User Guide](https://docs.aws.amazon.com/fsx/latest/LustreGuide/what-is.html). For details on how to deploy the Amazon FSx for Lustre CSI driver to your Amazon EKS cluster and verify that it works, see [Deploy the FSx for Lustre driver](fsx-csi-create.md). # Deploy the FSx for Lustre driver This topic shows you how to deploy the [FSx for Lustre CSI driver](fsx-csi.md) to your Amazon EKS cluster and verify that it works. We recommend using the latest version of the driver. For available versions, see [CSI Specification Compatibility Matrix](https://github.com/kubernetes-sigs/aws-fsx-csi-driver/blob/master/docs/README.md#csi-specification-compatibility-matrix) on GitHub. **Note** The driver isn’t supported on Fargate. For detailed descriptions of the available parameters and complete examples that demonstrate the driver’s features, see the [FSx for Lustre Container Storage Interface (CSI) driver](https://github.com/kubernetes-sigs/aws-fsx-csi-driver) project on GitHub. ## Prerequisites + An existing cluster. + The Amazon FSx CSI Driver EKS add-on supports authentication through either EKS Pod Identity or IAM Roles for Service Accounts (IRSA). To use EKS Pod Identity, install the Pod Identity agent before or after deploying the FSx CSI Driver add-on. For more information, see [Set up the Amazon EKS Pod Identity Agent](pod-id-agent-setup.md). To use IRSA instead, see [Create an IAM OIDC provider for your cluster](enable-iam-roles-for-service-accounts.md). + Version `2.12.3` or later or version `1.27.160` or later of the AWS Command Line Interface (AWS CLI) installed and configured on your device or AWS CloudShell. To check your current version, use `aws --version | cut -d / -f2 | cut -d ' ' -f1`. Package managers such as `yum`, `apt-get`, or Homebrew for macOS are often several versions behind the latest version of the AWS CLI. To install the latest version, see [Installing](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) and [Quick configuration with aws configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config) in the * AWS Command Line Interface User Guide*. The AWS CLI version that is installed in AWS CloudShell might also be several versions behind the latest version. To update it, see [Installing AWS CLI to your home directory](https://docs.aws.amazon.com/cloudshell/latest/userguide/vm-specs.html#install-cli-software) in the * AWS CloudShell User Guide*. + Version `0.215.0` or later of the `eksctl` command line tool installed on your device or AWS CloudShell. To install or update `eksctl`, see [Installation](https://eksctl.io/installation) in the `eksctl` documentation. + The `kubectl` command line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is `1.29`, you can use `kubectl` version `1.28`, `1.29`, or `1.30` with it. To install or upgrade `kubectl`, see [Set up `kubectl` and `eksctl`](install-kubectl.md). ## Step 1: Create an IAM role The Amazon FSx CSI plugin requires IAM permissions to make calls to AWS APIs on your behalf. **Note** Pods will have access to the permissions that are assigned to the IAM role unless you block access to IMDS. For more information, see [Secure Amazon EKS clusters with best practices](security-best-practices.md). The following procedure shows you how to create an IAM role and attach the AWS managed policy to it. 1. Create an IAM role and attach the AWS managed policy with the following command. Replace `my-cluster` with the name of the cluster you want to use. The command deploys an AWS CloudFormation stack that creates an IAM role and attaches the IAM policy to it. ``` eksctl create iamserviceaccount \ --name fsx-csi-controller-sa \ --namespace kube-system \ --cluster my-cluster \ --role-name AmazonEKS_FSx_CSI_DriverRole \ --role-only \ --attach-policy-arn arn:aws:iam::aws:policy/AmazonFSxFullAccess \ --approve ``` You’ll see several lines of output as the service account is created. The last lines of output are similar to the following. ``` [ℹ] 1 task: { 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/fsx-csi-controller-sa", create serviceaccount "kube-system/fsx-csi-controller-sa", } } [ℹ] building iamserviceaccount stack "eksctl-my-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa" [ℹ] deploying stack "eksctl-my-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa" [ℹ] waiting for CloudFormation stack "eksctl-my-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa" [ℹ] created serviceaccount "kube-system/fsx-csi-controller-sa" ``` Note the name of the AWS CloudFormation stack that was deployed. In the previous example output, the stack is named `eksctl-my-cluster-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa`. Now that you have created the Amazon FSx CSI driver IAM role, you can continue to the next section. When you deploy the add-on with this IAM role, it creates and is configured to use a service account that’s named `fsx-csi-controller-sa`. The service account is bound to a Kubernetes `clusterrole` that’s assigned the required Kubernetes permissions. ## Step 2: Install the Amazon FSx CSI driver We recommend that you install the Amazon FSx CSI driver through the Amazon EKS add-on to improve security and reduce the amount of work. To add an Amazon EKS add-on to your cluster, see [Create an Amazon EKS add-on](creating-an-add-on.md). For more information about add-ons, see [Amazon EKS add-ons](eks-add-ons.md). **Important** Pre-existing Amazon FSx CSI driver installations in the cluster can cause add-on installation failures. When you attempt to install the Amazon EKS add-on version while a non-EKS FSx CSI Driver exists, the installation will fail due to resource conflicts. Use the `OVERWRITE` flag during installation to resolve this issue. ``` aws eks create-addon --addon-name aws-fsx-csi-driver --cluster-name my-cluster --resolve-conflicts OVERWRITE ``` Alternatively, if you want a self-managed installation of the Amazon FSx CSI driver, see [Installation](https://github.com/kubernetes-sigs/aws-fsx-csi-driver/blob/master/docs/install.md) on GitHub. ## Step 3: Deploy a storage class, persistent volume claim, and sample app This procedure uses the [FSx for Lustre Container Storage Interface (CSI) driver](https://github.com/kubernetes-sigs/aws-fsx-csi-driver) GitHub repository to consume a dynamically-provisioned FSx for Lustre volume. 1. Note the security group for your cluster. You can see it in the AWS Management Console under the **Networking** section or by using the following AWS CLI command. Replace `my-cluster` with the name of the cluster you want to use. ``` aws eks describe-cluster --name my-cluster --query cluster.resourcesVpcConfig.clusterSecurityGroupId ``` 1. Create a security group for your Amazon FSx file system according to the criteria shown in [Amazon VPC Security Groups](https://docs.aws.amazon.com/fsx/latest/LustreGuide/limit-access-security-groups.html#fsx-vpc-security-groups) in the Amazon FSx for Lustre User Guide. For the **VPC**, select the VPC of your cluster as shown under the **Networking** section. For "the security groups associated with your Lustre clients", use your cluster security group. You can leave the outbound rules alone to allow **All traffic**. 1. Download the storage class manifest with the following command. ``` curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/storageclass.yaml ``` 1. Edit the parameters section of the `storageclass.yaml` file. Replace every example value with your own values. ``` parameters: subnetId: subnet-0eabfaa81fb22bcaf securityGroupIds: sg-068000ccf82dfba88 deploymentType: PERSISTENT_1 automaticBackupRetentionDays: "1" dailyAutomaticBackupStartTime: "00:00" copyTagsToBackups: "true" perUnitStorageThroughput: "200" dataCompressionType: "NONE" weeklyMaintenanceStartTime: "7:09:00" fileSystemTypeVersion: "2.12" ``` + ** `subnetId` ** – The subnet ID that the Amazon FSx for Lustre file system should be created in. Amazon FSx for Lustre isn’t supported in all Availability Zones. Open the Amazon FSx for Lustre console at https://console.aws.amazon.com/fsx/ to confirm that the subnet that you want to use is in a supported Availability Zone. The subnet can include your nodes, or can be a different subnet or VPC: + You can check for the node subnets in the AWS Management Console by selecting the node group under the **Compute** section. + If the subnet that you specify isn’t the same subnet that you have nodes in, then your VPCs must be [connected](https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/amazon-vpc-to-amazon-vpc-connectivity-options.html), and you must ensure that you have the necessary ports open in your security groups. + ** `securityGroupIds` ** – The ID of the security group you created for the file system. + ** `deploymentType` (optional)** – The file system deployment type. Valid values are `SCRATCH_1`, `SCRATCH_2`, `PERSISTENT_1`, and `PERSISTENT_2`. For more information about deployment types, see [Create your Amazon FSx for Lustre file system](https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started-step1.html). + **other parameters (optional)** – For information about the other parameters, see [Edit StorageClass](https://github.com/kubernetes-sigs/aws-fsx-csi-driver/tree/master/examples/kubernetes/dynamic_provisioning#edit-storageclass) on GitHub. 1. Create the storage class manifest. ``` kubectl apply -f storageclass.yaml ``` An example output is as follows. ``` storageclass.storage.k8s.io/fsx-sc created ``` 1. Download the persistent volume claim manifest. ``` curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/claim.yaml ``` 1. (Optional) Edit the `claim.yaml` file. Change `1200Gi` to one of the following increment values, based on your storage requirements and the `deploymentType` that you selected in a previous step. ``` storage: 1200Gi ``` + `SCRATCH_2` and `PERSISTENT` – `1.2 TiB`, `2.4 TiB`, or increments of 2.4 TiB over 2.4 TiB. + `SCRATCH_1` – `1.2 TiB`, `2.4 TiB`, `3.6 TiB`, or increments of 3.6 TiB over 3.6 TiB. 1. Create the persistent volume claim. ``` kubectl apply -f claim.yaml ``` An example output is as follows. ``` persistentvolumeclaim/fsx-claim created ``` 1. Confirm that the file system is provisioned. ``` kubectl describe pvc ``` An example output is as follows. ``` Name: fsx-claim Namespace: default StorageClass: fsx-sc Status: Bound [...] ``` **Note** The `Status` may show as `Pending` for 5-10 minutes, before changing to `Bound`. Don’t continue with the next step until the `Status` is `Bound`. If the `Status` shows `Pending` for more than 10 minutes, use warning messages in the `Events` as reference for addressing any problems. 1. Deploy the sample application. ``` kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/pod.yaml ``` 1. Verify that the sample application is running. ``` kubectl get pods ``` An example output is as follows. ``` NAME READY STATUS RESTARTS AGE fsx-app 1/1 Running 0 8s ``` 1. Verify that the file system is mounted correctly by the application. ``` kubectl exec -ti fsx-app -- df -h ``` An example output is as follows. ``` Filesystem Size Used Avail Use% Mounted on overlay 80G 4.0G 77G 5% / tmpfs 64M 0 64M 0% /dev tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup 192.0.2.0@tcp:/abcdef01 1.1T 7.8M 1.1T 1% /data /dev/nvme0n1p1 80G 4.0G 77G 5% /etc/hosts shm 64M 0 64M 0% /dev/shm tmpfs 6.9G 12K 6.9G 1% /run/secrets/kubernetes.io/serviceaccount tmpfs 3.8G 0 3.8G 0% /proc/acpi tmpfs 3.8G 0 3.8G 0% /sys/firmware ``` 1. Verify that data was written to the FSx for Lustre file system by the sample app. ``` kubectl exec -it fsx-app -- ls /data ``` An example output is as follows. ``` out.txt ``` This example output shows that the sample app successfully wrote the `out.txt` file to the file system. **Note** Before deleting the cluster, make sure to delete the FSx for Lustre file system. For more information, see [Clean up resources](https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started-step4.html) in the *FSx for Lustre User Guide*. ## Performance tuning for FSx for Lustre When using FSx for Lustre with Amazon EKS, you can optimize performance by applying Lustre tunings during node initialization. The recommended approach is to use launch template user data to ensure consistent configuration across all nodes. These tunings include: + Network and RPC optimizations + Lustre module management + LRU (Lock Resource Unit) tunings + Client cache control settings + RPC controls for OST and MDC For detailed instructions on implementing these performance tunings: + For optimizing performance for standard nodes (non-EFA), see [Optimize Amazon FSx for Lustre performance on nodes (non-EFA)](fsx-csi-tuning-non-efa.md) for a complete script that can be added to your launch template user data. + For optimizing performance for EFA-enabled nodes, see [Optimize Amazon FSx for Lustre performance on nodes (EFA)](fsx-csi-tuning-efa.md). # Optimize Amazon FSx for Lustre performance on nodes (EFA) This topic describes how to set up Elastic Fabric Adapter (EFA) tuning with Amazon EKS and Amazon FSx for Lustre. **Note** For information on creating and deploying the FSx for Lustre CSI driver, see [Deploy the FSx for Lustre driver](fsx-csi-create.md). For optimizing standard nodes without EFA, see [Optimize Amazon FSx for Lustre performance on nodes (non-EFA)](fsx-csi-tuning-non-efa.md). ## Step 1. Create EKS cluster Create a cluster using the provided configuration file: ``` # Create cluster using efa-cluster.yaml eksctl create cluster -f efa-cluster.yaml ``` Example `efa-cluster.yaml`: ``` #efa-cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: csi-fsx region: us-east-1 version: "1.30" iam: withOIDC: true availabilityZones: ["us-east-1a", "us-east-1d"] managedNodeGroups: - name: my-efa-ng instanceType: c6gn.16xlarge minSize: 1 desiredCapacity: 1 maxSize: 1 availabilityZones: ["us-east-1b"] volumeSize: 300 privateNetworking: true amiFamily: Ubuntu2204 efaEnabled: true preBootstrapCommands: - | #!/bin/bash eth_intf="$(/sbin/ip -br -4 a sh | grep $(hostname -i)/ | awk '{print $1}')" efa_version=$(modinfo efa | awk '/^version:/ {print $2}' | sed 's/[^0-9.]//g') min_efa_version="2.12.1" if [[ "$(printf '%s\n' "$min_efa_version" "$efa_version" | sort -V | head -n1)" != "$min_efa_version" ]]; then sudo curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.37.0.tar.gz tar -xf aws-efa-installer-1.37.0.tar.gz && cd aws-efa-installer echo "Installing EFA driver" sudo apt-get update && apt-get upgrade -y sudo apt install -y pciutils environment-modules libnl-3-dev libnl-route-3-200 libnl-route-3-dev dkms sudo ./efa_installer.sh -y modinfo efa else echo "Using EFA driver version $efa_version" fi echo "Installing Lustre client" sudo wget -O - https://fsx-lustre-client-repo-public-keys.s3.amazonaws.com/fsx-ubuntu-public-key.asc | gpg --dearmor | sudo tee /usr/share/keyrings/fsx-ubuntu-public-key.gpg > /dev/null sudo echo "deb [signed-by=/usr/share/keyrings/fsx-ubuntu-public-key.gpg] https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu jammy main" > /etc/apt/sources.list.d/fsxlustreclientrepo.list sudo apt update | tail sudo apt install -y lustre-client-modules-$(uname -r) amazon-ec2-utils | tail modinfo lustre echo "Loading Lustre/EFA modules..." sudo /sbin/modprobe lnet sudo /sbin/modprobe kefalnd ipif_name="$eth_intf" sudo /sbin/modprobe ksocklnd sudo lnetctl lnet configure echo "Configuring TCP interface..." sudo lnetctl net del --net tcp 2> /dev/null sudo lnetctl net add --net tcp --if $eth_intf # For P5 instance type which supports 32 network cards, # by default add 8 EFA interfaces selecting every 4th device (1 per PCI bus) echo "Configuring EFA interface(s)..." instance_type="$(ec2-metadata --instance-type | awk '{ print $2 }')" num_efa_devices="$(ls -1 /sys/class/infiniband | wc -l)" echo "Found $num_efa_devices available EFA device(s)" if [[ "$instance_type" == "p5.48xlarge" || "$instance_type" == "p5e.48xlarge" ]]; then for intf in $(ls -1 /sys/class/infiniband | awk 'NR % 4 == 1'); do sudo lnetctl net add --net efa --if $intf --peer-credits 32 done else # Other instances: Configure 2 EFA interfaces by default if the instance supports multiple network cards, # or 1 interface for single network card instances # Can be modified to add more interfaces if instance type supports it sudo lnetctl net add --net efa --if $(ls -1 /sys/class/infiniband | head -n1) --peer-credits 32 if [[ $num_efa_devices -gt 1 ]]; then sudo lnetctl net add --net efa --if $(ls -1 /sys/class/infiniband | tail -n1) --peer-credits 32 fi fi echo "Setting discovery and UDSP rule" sudo lnetctl set discovery 1 sudo lnetctl udsp add --src efa --priority 0 sudo /sbin/modprobe lustre sudo lnetctl net show echo "Added $(sudo lnetctl net show | grep -c '@efa') EFA interface(s)" ``` ## Step 2. Create node group Create an EFA-enabled node group: ``` # Create node group using efa-ng.yaml eksctl create nodegroup -f efa-ng.yaml ``` **Important** Adjust these values for your environment in section `# 5. Mount FSx filesystem`. ``` FSX_DNS="" # Needs to be adjusted. MOUNT_NAME="" # Needs to be adjusted. MOUNT_POINT="" # Needs to be adjusted. ``` Example `efa-ng.yaml`: ``` apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: final-efa region: us-east-1 managedNodeGroups: - name: ng-1 instanceType: c6gn.16xlarge minSize: 1 desiredCapacity: 1 maxSize: 1 availabilityZones: ["us-east-1a"] volumeSize: 300 privateNetworking: true amiFamily: Ubuntu2204 efaEnabled: true preBootstrapCommands: - | #!/bin/bash exec 1> >(logger -s -t $(basename $0)) 2>&1 ######################################################################################### # Configuration Section # ######################################################################################### # File System Configuration FSX_DNS="" # Needs to be adjusted. MOUNT_NAME="" # Needs to be adjusted. MOUNT_POINT="" # Needs to be adjusted. # Lustre Tuning Parameters LUSTRE_LRU_MAX_AGE=600000 LUSTRE_MAX_CACHED_MB=64 LUSTRE_OST_MAX_RPC=32 LUSTRE_MDC_MAX_RPC=64 LUSTRE_MDC_MOD_RPC=50 # File paths FUNCTIONS_SCRIPT="/usr/local/bin/lustre_functions.sh" TUNINGS_SCRIPT="/usr/local/bin/apply_lustre_tunings.sh" SERVICE_FILE="/etc/systemd/system/lustre-tunings.service" #EFA MIN_EFA_VERSION="2.12.1" # Function to check if a command was successful check_success() { if [ $? -eq 0 ]; then echo "SUCCESS: $1" else echo "FAILED: $1" return 1 fi } echo "********Starting FSx for Lustre configuration********" # 1. Install Lustre client if grep -q '^ID=ubuntu' /etc/os-release; then echo "Detected Ubuntu, proceeding with Lustre setup..." # Add Lustre repository sudo wget -O - https://fsx-lustre-client-repo-public-keys.s3.amazonaws.com/fsx-ubuntu-public-key.asc | sudo gpg --dearmor | sudo tee /usr/share/keyrings/fsx-ubuntu-public-key.gpg > /dev/null echo "deb [signed-by=/usr/share/keyrings/fsx-ubuntu-public-key.gpg] https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu jammy main" | sudo tee /etc/apt/sources.list.d/fsxlustreclientrepo.list sudo apt-get update sudo apt-get install -y lustre-client-modules-$(uname -r) sudo apt-get install -y lustre-client else echo "Not Ubuntu, exiting" exit 1 fi check_success "Install Lustre client" # Ensure Lustre tools are in the PATH export PATH=$PATH:/usr/sbin # 2. Apply network and RPC tunings echo "********Applying network and RPC tunings********" if ! grep -q "options ptlrpc ptlrpcd_per_cpt_max" /etc/modprobe.d/modprobe.conf; then echo "options ptlrpc ptlrpcd_per_cpt_max=64" | sudo tee -a /etc/modprobe.d/modprobe.conf check_success "Set ptlrpcd_per_cpt_max" else echo "ptlrpcd_per_cpt_max already set in modprobe.conf" fi if ! grep -q "options ksocklnd credits" /etc/modprobe.d/modprobe.conf; then echo "options ksocklnd credits=2560" | sudo tee -a /etc/modprobe.d/modprobe.conf check_success "Set ksocklnd credits" else echo "ksocklnd credits already set in modprobe.conf" fi # 3. Load Lustre modules manage_lustre_modules() { echo "Checking for existing Lustre modules..." if lsmod | grep -q lustre; then echo "Existing Lustre modules found." # Check for mounted Lustre filesystems echo "Checking for mounted Lustre filesystems..." if mount | grep -q "type lustre"; then echo "Found mounted Lustre filesystems. Attempting to unmount..." mounted_fs=$(mount | grep "type lustre" | awk '{print $3}') for fs in $mounted_fs; do echo "Unmounting $fs" sudo umount $fs check_success "Unmount filesystem $fs" done else echo "No Lustre filesystems mounted." fi # After unmounting, try to remove modules echo "Attempting to remove Lustre modules..." sudo lustre_rmmod if [ $? -eq 0 ]; then echo "SUCCESS: Removed existing Lustre modules" else echo "WARNING: Could not remove Lustre modules. They may still be in use." echo "Please check for any remaining Lustre processes or mounts." return 1 fi else echo "No existing Lustre modules found." fi echo "Loading Lustre modules..." sudo modprobe lustre check_success "Load Lustre modules" || exit 1 echo "Checking loaded Lustre modules:" lsmod | grep lustre } # Managing Lustre kernel modules echo "********Managing Lustre kernel modules********" manage_lustre_modules # 4. Initializing Lustre networking echo "********Initializing Lustre networking********" sudo lctl network up check_success "Initialize Lustre networking" || exit 1 # 4.5 EFA Setup and Configuration setup_efa() { echo "********Starting EFA Setup********" # Get interface and version information eth_intf="$(/sbin/ip -br -4 a sh | grep $(hostname -i)/ | awk '{print $1}')" efa_version=$(modinfo efa | awk '/^version:/ {print $2}' | sed 's/[^0-9.]//g') min_efa_version=$MIN_EFA_VERSION # Install or verify EFA driver if [[ "$(printf '%s\n' "$min_efa_version" "$efa_version" | sort -V | head -n1)" != "$min_efa_version" ]]; then echo "Installing EFA driver..." sudo curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.37.0.tar.gz tar -xf aws-efa-installer-1.37.0.tar.gz && cd aws-efa-installer # Install dependencies sudo apt-get update && apt-get upgrade -y sudo apt install -y pciutils environment-modules libnl-3-dev libnl-route-3-200 libnl-route-3-dev dkms # Install EFA sudo ./efa_installer.sh -y modinfo efa else echo "Using existing EFA driver version $efa_version" fi } configure_efa_network() { echo "********Configuring EFA Network********" # Load required modules echo "Loading network modules..." sudo /sbin/modprobe lnet sudo /sbin/modprobe kefalnd ipif_name="$eth_intf" sudo /sbin/modprobe ksocklnd # Initialize LNet echo "Initializing LNet..." sudo lnetctl lnet configure # Configure TCP interface echo "Configuring TCP interface..." sudo lnetctl net del --net tcp 2> /dev/null sudo lnetctl net add --net tcp --if $eth_intf # For P5 instance type which supports 32 network cards, # by default add 8 EFA interfaces selecting every 4th device (1 per PCI bus) echo "Configuring EFA interface(s)..." instance_type="$(ec2-metadata --instance-type | awk '{ print $2 }')" num_efa_devices="$(ls -1 /sys/class/infiniband | wc -l)" echo "Found $num_efa_devices available EFA device(s)" if [[ "$instance_type" == "p5.48xlarge" || "$instance_type" == "p5e.48xlarge" ]]; then # P5 instance configuration for intf in $(ls -1 /sys/class/infiniband | awk 'NR % 4 == 1'); do sudo lnetctl net add --net efa --if $intf --peer-credits 32 done else # Standard configuration # Other instances: Configure 2 EFA interfaces by default if the instance supports multiple network cards, # or 1 interface for single network card instances # Can be modified to add more interfaces if instance type supports it sudo lnetctl net add --net efa --if $(ls -1 /sys/class/infiniband | head -n1) --peer-credits 32 if [[ $num_efa_devices -gt 1 ]]; then sudo lnetctl net add --net efa --if $(ls -1 /sys/class/infiniband | tail -n1) --peer-credits 32 fi fi # Configure discovery and UDSP echo "Setting up discovery and UDSP rules..." sudo lnetctl set discovery 1 sudo lnetctl udsp add --src efa --priority 0 sudo /sbin/modprobe lustre # Verify configuration echo "Verifying EFA network configuration..." sudo lnetctl net show echo "Added $(sudo lnetctl net show | grep -c '@efa') EFA interface(s)" } # Main execution setup_efa configure_efa_network # 5. Mount FSx filesystem if [ ! -z "$FSX_DNS" ] && [ ! -z "$MOUNT_NAME" ]; then echo "********Creating mount point********" sudo mkdir -p $MOUNT_POINT check_success "Create mount point" echo "Mounting FSx filesystem..." sudo mount -t lustre ${FSX_DNS}@tcp:/${MOUNT_NAME} ${MOUNT_POINT} check_success "Mount FSx filesystem" else echo "Skipping FSx mount as DNS or mount name is not provided" fi # 6. Applying Lustre performance tunings echo "********Applying Lustre performance tunings********" # Get number of CPUs NUM_CPUS=$(nproc) # Calculate LRU size (100 * number of CPUs) LRU_SIZE=$((100 * NUM_CPUS)) #Apply LRU tunings echo "Apply LRU tunings" sudo lctl set_param ldlm.namespaces.*.lru_max_age=${LUSTRE_LRU_MAX_AGE} check_success "Set lru_max_age" sudo lctl set_param ldlm.namespaces.*.lru_size=$LRU_SIZE check_success "Set lru_size" # Client Cache Control sudo lctl set_param llite.*.max_cached_mb=${LUSTRE_MAX_CACHED_MB} check_success "Set max_cached_mb" # RPC Controls sudo lctl set_param osc.*OST*.max_rpcs_in_flight=${LUSTRE_OST_MAX_RPC} check_success "Set OST max_rpcs_in_flight" sudo lctl set_param mdc.*.max_rpcs_in_flight=${LUSTRE_MDC_MAX_RPC} check_success "Set MDC max_rpcs_in_flight" sudo lctl set_param mdc.*.max_mod_rpcs_in_flight=${LUSTRE_MDC_MOD_RPC} check_success "Set MDC max_mod_rpcs_in_flight" # 7. Verify all tunings echo "********Verifying all tunings********" # Function to verify parameter value verify_param() { local param=$1 local expected=$2 local actual=$3 if [ "$actual" == "$expected" ]; then echo "SUCCESS: $param is correctly set to $expected" else echo "WARNING: $param is set to $actual (expected $expected)" fi } echo "Verifying all parameters:" # LRU tunings actual_lru_max_age=$(lctl get_param -n ldlm.namespaces.*.lru_max_age | head -1) verify_param "lru_max_age" "600000" "$actual_lru_max_age" actual_lru_size=$(lctl get_param -n ldlm.namespaces.*.lru_size | head -1) verify_param "lru_size" "$LRU_SIZE" "$actual_lru_size" # Client Cache actual_max_cached_mb=$(lctl get_param -n llite.*.max_cached_mb | grep "max_cached_mb:" | awk '{print $2}') verify_param "max_cached_mb" "64" "$actual_max_cached_mb" # RPC Controls actual_ost_rpcs=$(lctl get_param -n osc.*OST*.max_rpcs_in_flight | head -1) verify_param "OST max_rpcs_in_flight" "32" "$actual_ost_rpcs" actual_mdc_rpcs=$(lctl get_param -n mdc.*.max_rpcs_in_flight | head -1) verify_param "MDC max_rpcs_in_flight" "64" "$actual_mdc_rpcs" actual_mdc_mod_rpcs=$(lctl get_param -n mdc.*.max_mod_rpcs_in_flight | head -1) verify_param "MDC max_mod_rpcs_in_flight" "50" "$actual_mdc_mod_rpcs" # Network and RPC configurations from modprobe.conf actual_ptlrpc=$(grep "ptlrpc ptlrpcd_per_cpt_max" /etc/modprobe.d/modprobe.conf | awk '{print $3}') verify_param "ptlrpcd_per_cpt_max" "ptlrpcd_per_cpt_max=64" "$actual_ptlrpc" actual_ksocklnd=$(grep "ksocklnd credits" /etc/modprobe.d/modprobe.conf | awk '{print $3}') verify_param "ksocklnd credits" "credits=2560" "$actual_ksocklnd" # 8. Setup persistence setup_persistence() { # Create functions file cat << EOF > $FUNCTIONS_SCRIPT #!/bin/bash apply_lustre_tunings() { local NUM_CPUS=\$(nproc) local LRU_SIZE=\$((100 * NUM_CPUS)) echo "Applying Lustre performance tunings..." lctl set_param ldlm.namespaces.*.lru_max_age=$LUSTRE_LRU_MAX_AGE lctl set_param ldlm.namespaces.*.lru_size=\$LRU_SIZE lctl set_param llite.*.max_cached_mb=$LUSTRE_MAX_CACHED_MB lctl set_param osc.*OST*.max_rpcs_in_flight=$LUSTRE_OST_MAX_RPC lctl set_param mdc.*.max_rpcs_in_flight=$LUSTRE_MDC_MAX_RPC lctl set_param mdc.*.max_mod_rpcs_in_flight=$LUSTRE_MDC_MOD_RPC } EOF # Create tuning script cat << EOF > $TUNINGS_SCRIPT #!/bin/bash exec 1> >(logger -s -t \$(basename \$0)) 2>&1 source $FUNCTIONS_SCRIPT # Function to check if Lustre is mounted is_lustre_mounted() { mount | grep -q "type lustre" } # Function to mount Lustre mount_lustre() { echo "Mounting Lustre filesystem..." mkdir -p $MOUNT_POINT mount -t lustre ${FSX_DNS}@tcp:/${MOUNT_NAME} $MOUNT_POINT return \$? } # Main execution # Try to mount if not already mounted if ! is_lustre_mounted; then echo "Lustre filesystem not mounted, attempting to mount..." mount_lustre fi # Wait for successful mount (up to 5 minutes) for i in {1..30}; do if is_lustre_mounted; then echo "Lustre filesystem mounted, applying tunings..." apply_lustre_tunings exit 0 fi echo "Waiting for Lustre filesystem to be mounted... (attempt $i/30)" sleep 10 done echo "Timeout waiting for Lustre filesystem mount" exit 1 EOF # Create systemd service # Create systemd directory if it doesn't exist sudo mkdir -p /etc/systemd/system/ # Create service file directly for Ubuntu cat << EOF > $SERVICE_FILE [Unit] Description=Apply Lustre Performance Tunings After=network.target remote-fs.target [Service] Type=oneshot ExecStart=/bin/bash -c 'source $FUNCTIONS_SCRIPT && $TUNINGS_SCRIPT' RemainAfterExit=yes [Install] WantedBy=multi-user.target EOF # Make scripts executable and enable service sudo chmod +x $FUNCTIONS_SCRIPT sudo chmod +x $TUNINGS_SCRIPT systemctl enable lustre-tunings.service systemctl start lustre-tunings.service } echo "********Setting up persistent tuning********" setup_persistence echo "FSx for Lustre configuration completed." ``` ## (Optional) Step 3. Verify EFA setup SSH into node: ``` # Get instance ID from EKS console or {aws} CLI ssh -i /path/to/your-key.pem ec2-user@ ``` Verify EFA configuration: ``` sudo lnetctl net show ``` Check setup logs: ``` sudo cat /var/log/cloud-init-output.log ``` Here’s example expected output for `lnetctl net show`: ``` net: - net type: tcp ... - net type: efa local NI(s): - nid: xxx.xxx.xxx.xxx@efa status: up ``` ## Example deployments ### a. Create claim.yaml ``` #claim.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: fsx-claim-efa spec: accessModes: - ReadWriteMany storageClassName: "" resources: requests: storage: 4800Gi volumeName: fsx-pv ``` Apply the claim: ``` kubectl apply -f claim.yaml ``` ### b. Create pv.yaml Update the ``: ``` #pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: fsx-pv spec: capacity: storage: 4800Gi volumeMode: Filesystem accessModes: - ReadWriteMany mountOptions: - flock persistentVolumeReclaimPolicy: Recycle csi: driver: fsx.csi.aws.com volumeHandle: fs-<1234567890abcdef0> volumeAttributes: dnsname: fs-<1234567890abcdef0>.fsx.us-east-1.amazonaws.com mountname: ``` Apply the persistent volume: ``` kubectl apply -f pv.yaml ``` ### c. Create pod.yaml ``` #pod.yaml apiVersion: v1 kind: Pod metadata: name: fsx-efa-app spec: containers: - name: app image: amazonlinux:2 command: ["/bin/sh"] args: ["-c", "while true; do dd if=/dev/urandom bs=100M count=20 > data/test_file; sleep 10; done"] resources: requests: vpc.amazonaws.com/efa: 1 limits: vpc.amazonaws.com/efa: 1 volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: fsx-claim-efa ``` Apply the Pod: ``` kubectl apply -f pod.yaml ``` ## Additional verification commands Verify Pod mounts and writes to filesystem: ``` kubectl exec -ti fsx-efa-app -- df -h | grep data # Expected output: # <192.0.2.0>@tcp:/ 4.5T 1.2G 4.5T 1% /data kubectl exec -ti fsx-efa-app -- ls /data # Expected output: # test_file ``` SSH onto the node to verify traffic is going over EFA: ``` sudo lnetctl net show -v ``` The expected output will show EFA interfaces with traffic statistics. ## Related information + [Deploy the FSx for Lustre driver](fsx-csi-create.md) + [Optimize Amazon FSx for Lustre performance on nodes (non-EFA)](fsx-csi-tuning-non-efa.md) + [Amazon FSx for Lustre Performance](https://docs.aws.amazon.com/fsx/latest/LustreGuide/performance.html) + [Elastic Fabric Adapter](https://docs.aws.amazon.com/ec2/latest/userguide/efa.html) # Optimize Amazon FSx for Lustre performance on nodes (non-EFA) You can optimize Amazon FSx for Lustre performance by applying tuning parameters during node initialization using launch template user data. **Note** For information on creating and deploying the FSx for Lustre CSI driver, see [Deploy the FSx for Lustre driver](fsx-csi-create.md). For optimizing performance with EFA-enabled nodes, see [Optimize Amazon FSx for Lustre performance on nodes (EFA)](fsx-csi-tuning-efa.md). ## Why use launch template user data? + Applies tunings automatically during node initialization. + Ensures consistent configuration across all nodes. + Eliminates the need for manual node configuration. ## Example script overview The example script defined in this topic performs these operations: ### `# 1. Install Lustre client` + Automatically detects your Amazon Linux (AL) OS version. + Installs the appropriate Lustre client package. ### `# 2. Apply network and RPC tunings` + Sets `ptlrpcd_per_cpt_max=64` for parallel RPC processing. + Configures `ksocklnd credits=2560` to optimize network buffers. ### `# 3. Load Lustre modules` + Safely removes existing Lustre modules if present. + Handles unmounting of existing filesystems. + Loads fresh Lustre modules. ### `# 4. Lustre Network Initialization` + Initializes Lustre networking configuration. + Sets up required network parameters. ### `# 5. Mount FSx filesystem` + You must adjust the values for your environment in this section. ### `# 6. Apply tunings` + LRU (Lock Resource Unit) tunings: + `lru_max_age=600000` + `lru_size` calculated based on CPU count + Client Cache Control: `max_cached_mb=64` + RPC Controls: + OST `max_rpcs_in_flight=32` + MDC `max_rpcs_in_flight=64` + MDC `max_mod_rpcs_in_flight=50` ### `# 7. Verify tunings` + Verifies all applied tunings. + Reports success or warning for each parameter. ### `# 8. Setup persistence` + You must adjust the values for your environment in this section as well. + Automatically detects your OS version (AL2023) to determine which `Systemd` service to apply. + System starts. + `Systemd` starts `lustre-tunings` service (due to `WantedBy=multi-user.target`). + Service runs `apply_lustre_tunings.sh` which: + Checks if filesystem is mounted. + Mounts filesystem if not mounted. + Waits for successful mount (up to five minutes). + Applies tuning parameters after successful mount. + Settings remain active until reboot. + Service exits after script completion. + Systemd marks service as "active (exited)". + Process repeats on next reboot. ## Create a launch template 1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/. 1. Choose **Launch Templates**. 1. Choose **Create launch template**. 1. In **Advanced details**, locate the **User data** section. 1. Paste the script below, updating anything as needed. **Important** Adjust these values for your environment in both section `# 5. Mount FSx filesystem` and the `setup_persistence()` function of `apply_lustre_tunings.sh` in section `# 8. Setup persistence`: ``` FSX_DNS="" # Needs to be adjusted. MOUNT_NAME="" # Needs to be adjusted. MOUNT_POINT="" # Needs to be adjusted. ``` ``` MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="==MYBOUNDARY==" --==MYBOUNDARY== Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash exec 1> >(logger -s -t $(basename $0)) 2>&1 # Function definitions check_success() { if [ $? -eq 0 ]; then echo "SUCCESS: $1" else echo "FAILED: $1" return 1 fi } apply_tunings() { local NUM_CPUS=$(nproc) local LRU_SIZE=$((100 * NUM_CPUS)) local params=( "ldlm.namespaces.*.lru_max_age=600000" "ldlm.namespaces.*.lru_size=$LRU_SIZE" "llite.*.max_cached_mb=64" "osc.*OST*.max_rpcs_in_flight=32" "mdc.*.max_rpcs_in_flight=64" "mdc.*.max_mod_rpcs_in_flight=50" ) for param in "${params[@]}"; do lctl set_param $param check_success "Set ${param%%=*}" done } verify_param() { local param=$1 local expected=$2 local actual=$3 if [ "$actual" == "$expected" ]; then echo "SUCCESS: $param is correctly set to $expected" else echo "WARNING: $param is set to $actual (expected $expected)" fi } verify_tunings() { local NUM_CPUS=$(nproc) local LRU_SIZE=$((100 * NUM_CPUS)) local params=( "ldlm.namespaces.*.lru_max_age:600000" "ldlm.namespaces.*.lru_size:$LRU_SIZE" "llite.*.max_cached_mb:64" "osc.*OST*.max_rpcs_in_flight:32" "mdc.*.max_rpcs_in_flight:64" "mdc.*.max_mod_rpcs_in_flight:50" ) echo "Verifying all parameters:" for param in "${params[@]}"; do name="${param%%:*}" expected="${param#*:}" actual=$(lctl get_param -n $name | head -1) verify_param "${name##*.}" "$expected" "$actual" done } setup_persistence() { # Create functions file cat << 'EOF' > /usr/local/bin/lustre_functions.sh #!/bin/bash apply_lustre_tunings() { local NUM_CPUS=$(nproc) local LRU_SIZE=$((100 * NUM_CPUS)) echo "Applying Lustre performance tunings..." lctl set_param ldlm.namespaces.*.lru_max_age=600000 lctl set_param ldlm.namespaces.*.lru_size=$LRU_SIZE lctl set_param llite.*.max_cached_mb=64 lctl set_param osc.*OST*.max_rpcs_in_flight=32 lctl set_param mdc.*.max_rpcs_in_flight=64 lctl set_param mdc.*.max_mod_rpcs_in_flight=50 } EOF # Create tuning script cat << 'EOF' > /usr/local/bin/apply_lustre_tunings.sh #!/bin/bash exec 1> >(logger -s -t $(basename $0)) 2>&1 # Source the functions source /usr/local/bin/lustre_functions.sh # FSx details FSX_DNS="" # Needs to be adjusted. MOUNT_NAME="" # Needs to be adjusted. MOUNT_POINT="" # Needs to be adjusted. # Function to check if Lustre is mounted is_lustre_mounted() { mount | grep -q "type lustre" } # Function to mount Lustre mount_lustre() { echo "Mounting Lustre filesystem..." mkdir -p ${MOUNT_POINT} mount -t lustre ${FSX_DNS}@tcp:/${MOUNT_NAME} ${MOUNT_POINT} return $? } # Main execution # Try to mount if not already mounted if ! is_lustre_mounted; then echo "Lustre filesystem not mounted, attempting to mount..." mount_lustre fi # Wait for successful mount (up to 5 minutes) for i in {1..30}; do if is_lustre_mounted; then echo "Lustre filesystem mounted, applying tunings..." apply_lustre_tunings exit 0 fi echo "Waiting for Lustre filesystem to be mounted... (attempt $i/30)" sleep 10 done echo "Timeout waiting for Lustre filesystem mount" exit 1 EOF # Create systemd service cat << 'EOF' > /etc/systemd/system/lustre-tunings.service [Unit] Description=Apply Lustre Performance Tunings After=network.target remote-fs.target StartLimitIntervalSec=0 [Service] Type=oneshot ExecStart=/usr/local/bin/apply_lustre_tunings.sh RemainAfterExit=yes Restart=on-failure RestartSec=30 [Install] WantedBy=multi-user.target EOF chmod +x /usr/local/bin/lustre_functions.sh chmod +x /usr/local/bin/apply_lustre_tunings.sh systemctl enable lustre-tunings.service systemctl start lustre-tunings.service } echo "Starting FSx for Lustre configuration..." # 1. Install Lustre client if grep -q 'VERSION="2"' /etc/os-release; then amazon-linux-extras install -y lustre elif grep -q 'VERSION="2023"' /etc/os-release; then dnf install -y lustre-client fi check_success "Install Lustre client" # 2. Apply network and RPC tunings export PATH=$PATH:/usr/sbin echo "Applying network and RPC tunings..." if ! grep -q "options ptlrpc ptlrpcd_per_cpt_max" /etc/modprobe.d/modprobe.conf; then echo "options ptlrpc ptlrpcd_per_cpt_max=64" | tee -a /etc/modprobe.d/modprobe.conf echo "options ksocklnd credits=2560" | tee -a /etc/modprobe.d/modprobe.conf fi # 3. Load Lustre modules modprobe lustre check_success "Load Lustre modules" || exit 1 # 4. Lustre Network Initialization lctl network up check_success "Initialize Lustre networking" || exit 1 # 5. Mount FSx filesystem FSX_DNS="" # Needs to be adjusted. MOUNT_NAME="" # Needs to be adjusted. MOUNT_POINT="" # Needs to be adjusted. if [ ! -z "$FSX_DNS" ] && [ ! -z "$MOUNT_NAME" ]; then mkdir -p $MOUNT_POINT mount -t lustre ${FSX_DNS}@tcp:/${MOUNT_NAME} ${MOUNT_POINT} check_success "Mount FSx filesystem" fi # 6. Apply tunings apply_tunings # 7. Verify tunings verify_tunings # 8. Setup persistence setup_persistence echo "FSx for Lustre configuration completed." --==MYBOUNDARY==-- ``` 1. When creating Amazon EKS node groups, select this launch template. For more information, see [Create a managed node group for your cluster](create-managed-node-group.md). ## Related information + [Deploy the FSx for Lustre driver](fsx-csi-create.md) + [Optimize Amazon FSx for Lustre performance on nodes (EFA)](fsx-csi-tuning-efa.md) + [Amazon FSx for Lustre Performance](https://docs.aws.amazon.com/fsx/latest/LustreGuide/performance.html)