Convert AWS ParallelCluster managed storage to external storage - AWS ParallelCluster

Convert AWS ParallelCluster managed storage to external storage

Learn how to convert AWS ParallelCluster managed storage to external storage.

The procedures are based on the following example configuration file snippet.

...
  - MountDir: /fsx
    Name: fsx
    StorageType: FsxLustre
    FsxLustreSettings:
      StorageCapacity: 1200
      DeletionPolicy: Delete
...
Convert AWS ParallelCluster managed storage to external storage

  1. Set the DeletionPolicy to Retain in the cluster configuration file.

    ...
   - MountDir: /fsx
     Name: fsx
     StorageType: FsxLustre
     FsxLustreSettings:
       StorageCapacity: 1200
       DeletionPolicy: Retain
...

  2. To set the DeletionPolicy change, run the following command.

    pcluster update-cluster -n cluster-name -c cluster-config.yaml

  3. Remove the SharedStorage section from the cluster configuration file.

    ...
...

  4. To change the managed SharedStorage to external SharedStorage and detach it from the cluster, run the following command.

    pcluster update-cluster -n cluster-name -c cluster-config.yaml

  5. Your shared storage is now external and detached from the cluster.

  6. To attach your external file system to the original cluster or another cluster, follow these steps.

    1. Get the FSx for Lustre file system ID.

      1. To use the AWS CLI run the following command and find the file system with a name that includes the name of your original cluster and note the file system ID.

        aws fsx describe-file-systems

      2. To use the AWS Management Console, log in and navigate to the https://console.aws.amazon.com/fsx/. In the list of file systems, find the file system with a name that includes the name of your original cluster and note the file system ID.

    2. Update the file system security group rules to provide access to and from the file system and cluster subnets. You can find the file system security group name and ID in the Amazon FSx console.

      Add rules to the file system security group that allow inbound and outbound TCP traffic from and to the head node and the compute node IP CIDR ranges or prefixes. Specify TCP ports 988, 1021, 1022, and 1023 for the inbound and outbound TCP traffic.

      For more information, see SharedStorage / FsxLustreSettings / FileSystemId and Creating, configuring, and deleting security groups for Amazon EC2 in the AWS Command Line Interface User Guide for Version 2.

    3. Add the SharedStorage section to the cluster configuration.

      ...
   - MountDir: /fsx
     Name: fsx-external
     StorageType: FsxLustre
     FsxLustreSettings:
       FileSystemId: fs-02e5b4b4abd62d51c
...

    4. To add the external shared storage to the cluster, run the following command.

      pcluster update-cluster -n cluster-name -c cluster-config.yaml