Performance guidelines and design patterns for S3 Express One Zone

When building applications that upload and retrieve objects from Amazon S3 Express One Zone, follow our best practice guidelines to optimize performance. To use the S3 Express One Zone storage class, you must create an S3 directory bucket. The S3 Express One Zone storage class isn't supported for use with S3 general purpose buckets.

For performance guidelines for all other Amazon S3 storage classes and S3 general purpose buckets, see Best practices design patterns: optimizing Amazon S3 performance.

To obtain the best performance for your application when using the S3 Express One Zone storage class and directory buckets, we recommend the following guidelines and design patterns.

Co-locate S3 Express One Zone storage with your AWS compute resources

Each directory bucket is stored in a single Availability Zone that you select when you create the bucket. You can get started by creating a new directory bucket in an Availability Zone local to your compute workloads or resources. You can then immediately begin very low-latency reads and writes. Directory buckets are the first S3 buckets where you can choose the Availability Zone in an AWS Region to reduce latency between compute and storage.

If you access directory buckets across Availability Zones, latency will increase. To optimize performance, we recommend that you access a directory bucket from Amazon Elastic Container Service, Amazon Elastic Kubernetes Service, and Amazon Elastic Compute Cloud instances that are located in the same Availability Zone when possible.

Directory buckets

Each directory bucket can support hundreds of thousands of transactions per second (TPS). Unlike general purpose buckets, directory buckets organize keys hierarchically into directories instead of prefixes. A prefix is a string of characters at the beginning of the object key name. You can think of prefixes as a way to organize your data in a similar way to directories. However, prefixes are not directories.

Prefixes organize data in a flat namespace within general purpose buckets, and there are no limits to the number of prefixes within a general purpose bucket. Each prefix can achieve at least 3,500 PUT/POST/DELETE or 5,500 GET/HEAD requests per second. You can also parallelize requests across multiple prefixes to scale performance. However, this scaling, in the case of both read and write operations, happens gradually and is not instantaneous. While general purpose buckets are scaling to your new higher request rate, you might receive some HTTP status code 503 (Service Unavailable) errors.

With a hierarchical namespace, the delimiter in the object key is important. The only supported delimiter is a forward slash (/). Directories are determined by delimiter boundaries. For example, the object key dir1/dir2/file1.txt results in the directories dir1/ and dir2/ being automatically created, and the object file1.txt being added to the /dir2 directory in the path dir1/dir2/file1.txt.

The directories that are created when objects are uploaded to directory buckets have no per-prefix TPS limits and are automatically pre-scaled to reduce the chance of HTTP 503 (Service Unavailable) errors. This automatic scaling allows your applications to parallelize read and write requests within and across directories as needed.

Directory bucket horizontal scaling request parallelization

You can achieve the best performance by issuing multiple concurrent requests to directory buckets to spread your requests over separate connections to maximize the accessible bandwidth. S3 Express One Zone doesn't have any limits for the number of connections made to your directory bucket. Individual directories can scale performance horizontally and automatically when large numbers of concurrent writes to the same directory are happening.

When an object key is initially created and its key name includes a directory, the directory is automatically created for the object. Subsequent object uploads to that same directory do not require the directory to be created, which reduces latency on object uploads to existing directories.

Although both shallow and deep directory structures are supported for storing objects within a directory bucket, directory buckets do automatically scale horizontally, with lower latency on concurrent uploads to the same directory or to parallel directory siblings.

Use session-based authentication

S3 Express One Zone and directory buckets support a new session-based authorization mechanism to authenticate and authorize requests to a directory bucket. With session-based authentication, the AWS SDKs automatically use the CreateSession API operation to create a temporary session token that can be used for low-latency authorization of data requests to a directory bucket.

The AWS SDKs use the CreateSession API operation to request temporary credentials, and then automatically create and refresh tokens for you on your behalf every 5 minutes. To take advantage of the performance benefits of the S3 Express One Zone storage class, we recommended that you use the AWS SDKs to initiate and manage the CreateSession API request. For more information about this session-based model, see Authorizing Zonal endpoint API operations with CreateSession.

S3 additional checksum best practices

S3 Express One Zone offers you the option to choose the checksum algorithm that is used to validate your data during upload or download. You can select one of the following Secure Hash Algorithms (SHA) or Cyclic Redundancy Check (CRC) data-integrity check algorithms: CRC32, CRC32C, SHA-1, and SHA-256. MD5-based checksums are not supported with the S3 Express One Zone storage class.

CRC32 is the default checksum used by the AWS SDKs when transmitting data to or from S3 Express One Zone. We recommend using CRC32 and CRC32C for the best performance with the S3 Express One Zone storage class.

Use the latest version of the AWS SDKs and common runtime libraries

Several of the AWS SDKs also provide the AWS Common Runtime (CRT) libraries to further accelerate performance in S3 clients. These SDKs include the AWS SDK for Java 2.x, the AWS SDK for C++, and the AWS SDK for Python (Boto3). The CRT-based S3 client transfers objects to and from S3 Express One Zone with enhanced performance and reliability by automatically using the multipart upload API operation and byte-range fetches to automate horizontally scaling connections.

To achieve the highest performance with the S3 Express One Zone storage class, we recommend using the latest version of the AWS SDKs that include the CRT libraries or using the AWS Command Line Interface (AWS CLI).

Performance troubleshooting

Retry requests for latency-sensitive applications

S3 Express One Zone is purpose-built to deliver consistent levels of high-performance without additional tuning. However, setting aggressive timeout values and retries can further help drive consistent latency and performance. The AWS SDKs have configurable timeout and retry values that you can tune to the tolerances of your specific application.

AWS Common Runtime (CRT) libraries and Amazon EC2 instance type pairing

Applications that perform a large number of read and write operations likely need more memory or computing capacity than applications that don't. When launching your Amazon Elastic Compute Cloud (Amazon EC2) instances for your performance-demanding workload, choose instance types that have the amount of these resources that your application needs. S3 Express One Zone high-performance storage is ideally paired with larger and newer instance types with larger amounts of system memory and more powerful CPUs and GPUs that can take advantage of higher-performance storage. We also recommend using the latest versions of the CRT-enabled AWS SDKs, which can better accelerate read and write requests in parallel.

Use session-based authentication in AWS SDKs instead of the HTTP REST APIs

With Amazon S3, you can also optimize performance when you're using HTTP REST API requests by following the same best practices that are part of the AWS SDKs. However, with the session-based authorization and authentication mechanism that's used by S3 Express One Zone, we strongly recommend that you use the AWS SDKs to manage CreateSession and its managed session token. The AWS SDKs automatically create and refresh tokens on your behalf by using the CreateSession API operation. Using CreateSession saves on per-request round-trip latency to AWS Identity and Access Management (IAM) to authorize each request.