Lustre data compression

You can use the Lustre data compression feature to achieve cost savings on your high-performance Amazon FSx for Lustre file systems and backup storage. When data compression is enabled, Amazon FSx for Lustre automatically compresses newly-written files before they are written to disk and automatically uncompresses them when they are read.

Data compression uses the LZ4 algorithm, which is optimized to deliver high levels of compression without adversely impacting file system performance. LZ4 is a Lustre community-trusted and performance-oriented algorithm that provides a balance between compression speed and compressed file size. Enabling data compression does not typically have a measurable impact on latency.

Data compression reduces the amount of data that is transferred between Amazon FSx for Lustre file servers and storage. If you are not already using compressed file formats, you will see an increase in overall file system throughput capacity when using data compression. Increases in throughput capacity that are related to data compression will be capped after you have saturated your front-end network interface cards.

For example, if your file system is a PERSISTENT-50 SSD deployment type, your network throughput has a baseline of 250 MB/s per TiB of storage. Your disk throughput has a baseline of 50 MB/s per TiB. With data compression, your disk throughput could increase from 50 MB/s per TiB to a maximum of 250 MB/s per TiB, which is the baseline network throughput limit. For more information about network and disk throughput limits, see the file system performance tables in Aggregate file system performance. For more information about data compression performance, see the Spend less while increasing performance with Amazon FSx for Lustre data compression post on the AWS Storage Blog.

Managing data compression

You can turn data compression on or off when creating a new Amazon FSx for Lustre file system. Data compression is turned off by default when you create an Amazon FSx for Lustre file system from the console, AWS CLI, or API.

{

    "FileSystems": [
        {
            "OwnerId": "111122223333",
            "CreationTime": 1549310341.483,
            "FileSystemId": "fs-0123456789abcdef0",
            "FileSystemType": "LUSTRE",
            "FileSystemTypeVersion": "2.12",
            "Lifecycle": "CREATING",
            "StorageCapacity": 3600,
            "VpcId": "vpc-123456",
            "SubnetIds": [
                "subnet-123456"
            ],
            "NetworkInterfaceIds": [
                "eni-039fcf55123456789"
            ],
            "DNSName": "fs-0123456789abcdef0.fsx.us-east-2.amazonaws.com",
            "ResourceARN": "arn:aws:fsx:us-east-2:123456:file-system/fs-0123456789abcdef0",
            "Tags": [
                {
                    "Key": "Name",
                    "Value": "Lustre-TEST-1"
                }
            ],
            "LustreConfiguration": {
                "DeploymentType": "PERSISTENT_1",
                "DataCompressionType": "LZ4",
                "PerUnitStorageThroughput": 50
            }
        }
    ]
}

You can also change the data compression configuration of your existing file systems. When you turn data compression on for an existing file system, only newly written files are compressed, and existing files are not compressed. For more information, see Compressing previously written files.

$ aws fsx update-file-system \
    --file-system-id fs-0123456789abcdef0 \
    --lustre-configuration DataCompressionType=LZ4

Data compression configuration when creating a file system from backup

You can use an available backup to create a new Amazon FSx for Lustre file system. When you create a new file system from backup, there is no need to specify the DataCompressionType; the setting will be applied using the backup's DataCompressionType setting. If you choose to specify the DataCompressionType when creating from backup, the value must match the backup's DataCompressionType setting.

To view the settings on a backup, choose it from the Backups tab of the Amazon FSx console. Details of the backup will be listed on the Summary page for the backup. You can also run the describe-backups AWS CLI command (the equivalent API action is DescribeBackups).

Compressing previously written files

Files are uncompressed if they were created when data compression was turned off on the Amazon FSx for Lustre file system. Turning on data compression will not automatically compress your existing uncompressed data.

You can use the lfs_migrate command that is installed as a part of the Lustre client installation to compress existing files. For an example, see FSxL-Compression which is available on GitHub.

Viewing file sizes

You can use the following commands to view the uncompressed and compressed sizes of your files and directories.

The following examples show the output of each command with the same file.

$ du -sh samplefile
272M	samplefile
$ du -sh --apparent-size samplefile
1.0G	samplefile
$ ls -lh samplefile
-rw-r--r-- 1 root root 1.0G May 10 21:16 samplefile

The -h option is useful for these commands because it prints sizes in a human-readable format.

Using CloudWatch metrics

You can use Amazon CloudWatch Logs metrics to view your file system usage. The LogicalDiskUsage metric shows the total logical disk usage (without compression), and the PhysicalDiskUsage metric shows the total physical disk usage (with compression). These two metrics are available only if your file system has data compression enabled or previously had it enabled.

You can determine your file system's compression ratio by dividing the Sum of the LogicalDiskUsage statistic by the Sum of the PhysicalDiskUsage statistic.

For more information about monitoring your file system’s performance, see Monitoring Amazon FSx for Lustre file systems.