Differences for directory buckets - Amazon Simple Storage Service

Differences for directory buckets

When using Amazon S3, you can choose the bucket type that best fits your application and performance requirements. A directory bucket is a type of bucket that is best used for low-latency use cases. You can create a directory bucketin a single Availability Zone to store data.You can create up to 100 directory buckets in each of your AWS accounts, with no limit on the number of objects that you can store in a bucket. Your bucket quota is applied to each Region in your AWS account. If your application requires increasing this limit, contact AWS Support. Directory buckets support the S3 Express One Zone storage class. The S3 Express One Zone storage class is recommended if your application is performance sensitive and benefits from single-digit millisecond PUT and GET latencies. To learn more about the S3 Express One Zone storage class, see S3 Express One Zone.

For more information about how directory buckets are different, see the following topics.

Differences for directory buckets

  • Directory bucket names

    • A directory bucket name consists of a base name that you provide and a suffix that contains the ID of the Zone (Availability Zone) that your bucket is located in. Directory bucket names must use a specific format and follow the naming rules for directory buckets. For a list of rules and examples of directory bucket names, see Directory bucket naming rules.

  • ListObjectsV2 behavior

    • For directory buckets, ListObjectsV2 does not return objects in lexicographical (alphabetical) order. Additionally, prefixes must end in a delimiter and only "/" can be specified as the delimiter.

    • For directory buckets, ListObjectsV2 response includes the prefixes that are related only to in-progress multipart uploads.

  • Deletion behavior – When you delete an object in a directory bucket, Amazon S3 recursively deletes any empty directories in the object path. For example, if you delete the object key dir1/dir2/file1.txt, Amazon S3 deletes file1.txt. If the dir1/ and dir2/ directories are empty and contain no other objects, Amazon S3 also deletes those directories.

  • ETags and checksums – Entity tags (ETags) for S3 Express One Zone are random alphanumeric strings unique to the object and not MD5 checksums. For more information about using additional checksums with S3 Express One Zone, see S3 additional checksum best practices.

  • Object keys in DeleteObjects requests

    • Object keys in DeleteObjects requests must contain at least one non-white space character. Strings of all white space characters aren't supported in DeleteObjects requests.

    • Object keys in DeleteObjects requests cannot contain Unicode control characters, except for the newline (\n), tab (\t), and carriage return (\r) characters.

  • Regional and Zonal endpoints – Bucket-management API operations for directory buckets are available through a Regional endpoint and are referred to as Regional endpoint API operations. Examples of Regional endpoint API operations are CreateBucket and DeleteBucket. After you create a directory bucket, you can use Zonal endpoint API operations to upload and manage the objects in your directory bucket. Zonal endpoint API operations are available through a Zonal endpoint. Examples of Zonal endpoint API operations are PutObject and CopyObject. When using directory buckets, you must specify the Region in all requests. For Regional endpoints, you specify the Region, for example, s3express-control.us-west-2.amazonaws.com. For Zonal endpoints, you specify both the Region and the Availability Zone, for example, s3express-usw2-az1.us-west-2.amazonaws.com. For more information, see Regional and Zonal endpoints for directory buckets.

  • Multipart uploads – You can upload and copy large objects that are stored in directory buckets by using the multipart upload process. However, the following are some differences when using the multipart upload process with objects stored in directory buckets. For more information, see Using multipart uploads with directory buckets.

    • The object creation date is the completion date of the multipart upload.

    • Multipart part numbers must use consecutive part numbers. If you try to complete a multipart upload request with nonconsecutive part numbers, Amazon S3 generates an HTTP 400 (Bad Request) error.

    • The initiator of a multipart upload can abort the multipart upload request only if they have been granted explicit allow access to AbortMultipartUpload through the s3express:CreateSession permission. For more information, see Authorizing Regional endpoint API operations with IAM.

  • Emptying a directory bucket – The s3 rm command through the AWS Command Line Interface (CLI), the delete operation through Mountpoint, and the Empty bucket option button through the AWS Management Console are unable to delete in-progress multipart uploads in a directory bucket. To delete these in-progress multipart uploads, use the ListMultipartUploads operation to list the in-progress multipart uploads in the bucket and use the AbortMultipartUpload operation to abort all the in-progress multipart uploads.

  • Server-side encryption

    • For directory buckets, to encrypt your data with server-side encryption, you can use either server-side encryption with Amazon S3 managed keys (SSE-S3) (the default) or server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). We recommend that the bucket's default encryption uses the desired encryption configurations and you don't override the bucket default encryption in your CreateSession requests or PUT object requests. Then, new objects are automatically encrypted with the desired encryption settings. For more information about the encryption overriding behaviors in directory buckets, see Specifying server-side encryption with AWS KMS for new object uploads.

    • Your SSE-KMS configuration can only support 1 customer managed key per directory bucket for the lifetime of the bucket. The AWS managed key (aws/s3) isn't supported. Also, after you specify a customer managed key for SSE-KMS, you can't override the customer managed key for the bucket's SSE-KMS configuration.

      You can identify the customer managed key you specified for the bucket's SSE-KMS configuration, in the following way:

      • You make a HeadObject API operation request to find the value of x-amz-server-side-encryption-aws-kms-key-id in your response.

      To use a new customer managed key for your data, we recommend copying your existing objects to a new directory bucket with a new customer managed key.

    • For Zonal endpoint (object-level) API operations except CopyObject and UploadPartCopy, you authenticate and authorize requests through CreateSession for low latency. We recommend that the bucket's default encryption uses the desired encryption configuration and you don't override the bucket default encryption in your CreateSession requests or PUT object requests. Then, new objects are automatically encrypted with the desired encryption settings. To encrypt new objects in a directory bucket with SSE-KMS, you must specify SSE-KMS as the directory bucket's default encryption configuration with an KMS key (specifically, a customer managed key). Then, when a session is created for Zonal endpoint API operations, new objects are automatically encrypted and decrypted with SSE-KMS and S3 Bucket Keys during the session. For more information about the encryption overriding behaviors in directory buckets, see Specifying server-side encryption with AWS KMS for new object uploads.

    • For CopyObject, to encrypt new object copies in a directory bucket with SSE-KMS, you must specify SSE-KMS as the directory bucket's default encryption configuration with a KMS key (specifically, a customer managed key). Then, when you specify server-side encryption settings for new object copies with SSE-KMS, you must make sure the encryption key is the same customer managed key that you specified for the directory bucket's default encryption configuration. For UploadPartCopy, to encrypt new object part copies in a directory bucket with SSE-KMS, you must specify SSE-KMS as the directory bucket's default encryption configuration with a KMS key (specifically, a customer managed key). You can't specify server-side encryption settings for new object part copies with SSE-KMS in the UploadPartCopy request headers. Also, the encryption settings that you provide in the CreateMultipartUpload request must match the default encryption configuration of the destination bucket.

    • S3 Bucket Keys are always enabled for GET and PUT operations in a directory bucket and can’t be disabled. S3 Bucket Keys aren't supported, when you copy SSE-KMS encrypted objects from general purpose buckets to directory buckets, from directory buckets to general purpose buckets, or between directory buckets, through CopyObject, UploadPartCopy, the Copy operation in Batch Operations, or the import jobs. In this case, Amazon S3 makes a call to AWS KMS every time a copy request is made for a KMS-encrypted object.

    • When you specify an AWS KMS customer managed key for encryption in your directory bucket, only use the key ID or key ARN. The key alias format of the KMS key isn't supported.

API operations supported for directory buckets

The Amazon S3 Express One Zone storage class supports both Regional (bucket level, or control plane) and Zonal (object level, or data plane) endpoint API operations. For more information, see Networking for directory buckets and Endpoints and gateway VPC endpoints.

Regional endpoint API operations

The following Regional endpoint API operations are supported for S3 Express One Zone:

Zonal endpoint API operations

The following Zonal endpoint API operations are supported for use with directory buckets:

Amazon S3 features not supported by directory buckets

The following Amazon S3 features are not supported by S3 Express One Zone:

  • AWS managed policies

  • AWS PrivateLink for S3

  • MD5 checksums

  • Multi-factor authentication (MFA) delete

  • S3 Object Lock

  • Requester Pays

  • S3 Access Grants

  • S3 Access Points

  • Bucket tags

  • Amazon CloudWatch request metrics

  • S3 Event Notifications

  • S3 Lifecycle

  • S3 Multi-Region Access Points

  • S3 Object Lambda Access Points

  • S3 Versioning

  • S3 Inventory

  • S3 Replication

  • Object tags

  • S3 Select

  • Server access logs

  • Static website hosting

  • S3 Storage Lens

  • S3 Storage Lens groups

  • S3 Transfer Acceleration

  • Dual-layer server-side encryption with AWS Key Management Service (AWS KMS) keys (DSSE-KMS)

  • Server-side encryption with customer-provided keys (SSE-C)

  • The option to copy an existing bucket's settings when creating a new bucket in the Amazon S3 console

  • Enhanced access denied (HTTP 403 Forbidden) error messages