S3 Metadata tables schema

Note

The S3 Metadata feature is in preview release for Amazon S3 and is subject to change.

Amazon S3 metadata tables contain rows and columns. Each row represents a mutation event that has created, updated, or deleted an object in your general purpose bucket. Most of these events are the result of various user actions, but some of these events are the result of actions taken by Amazon S3 on your behalf, such as S3 Lifecycle expirations or storage class transitions.

S3 Metadata is an event-processing pipeline that is designed to keep the metadata table eventually consistent with what changes have occurred in your general purpose bucket. Be aware that by the time that S3 Metadata is notified that an object is created or updated, that object might already have been overwritten or deleted in the bucket. By default, a table row is created for each S3 bucket operation. However, if any object metadata is deleted or overwritten, or objects can no longer be retrieved, some columns might show a NULL value to indicate any missing metadata schema.

The following is an example of a metadata table for a general purpose bucket named amzn-s3-demo-bucket:


bucket                key                        sequence_number                                                                                          record_type   record_timestamp           version_id   is_delete_marker   size   last_modified_date   e_tag	                           storage_class  is_multipart   encryption_status   is_bucket_key_enabled   kms_key_arn                                                                   checksum_algorithm   object_tags   user_metadata	                                                                                                                 requester      source_ip_address   request_id 
amzn-s3-demo-bucket   Finance/statement1.pdf     80e737d8b4d82f776affffffffffffffff006737d8b4d82f776a00000000000000000000000000000000000000000000000072   CREATE        2024-11-15 23:26:44.899                 FALSE              6223   11/15/2024 23:26     e131b86632dda753aac4018f72192b83    STANDARD	  FALSE          SSE-KMS             FALSE                   arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890df   SSECRC32             {}            {count -> Asia, customs -> false, family -> true, location -> Mary, name -> football, user -> United States}                       111122223333   192.0.2.1           CVK8FWYRW0M9JW65
amzn-s3-demo-bucket   s3-dg.pdf                  80e737d8b4e39f1dbdffffffffffffffff006737d8b4e39f1dbd00000000000000000000000000000000000000000000000072   CREATE        2024-11-15 23:26:44.942                 FALSE              3554   11/15/2024 23:26     9bb49efc2d92c05558ddffbbde8636d5    STANDARD	  FALSE          DSSE-KMS            FALSE                   arn:aws:kms:us-east-1:936810216292:key/0dcebce6-49fd-4cae-b2e2-5512ad281afd   SSESHA1              {}            {}                                                                                                                                 111122223333   192.0.2.1           CVKAQDRAZEG7KXAY
amzn-s3-demo-bucket   Development/Projects.xls   80e737d8b4ed9ac5c6ffffffffffffffff006737d8b4ed9ac5c600000000000000000000000000000000000000000000000072   CREATE        2024-11-15 23:26:44.966                 FALSE              7746   11/15/2024 23:26     729a6863e47fb9955b31bfabce984908    STANDARD	  FALSE          SSE-S3              FALSE                   NULL                                                                          SSECRC32             {}            {count -> Asia, customs -> Canada, family -> Billiards, filter -> true, location -> Europe, name -> Asia, user -> United States}   111122223333   192.0.2.1           CVK7Z6XQTQ90BSRV

Metadata tables have the following schema:

Column name	Required?	Data type
`bucket`	Yes	String	The general purpose bucket name. For more information, see General purpose bucket naming rules.
`key`	Yes	String	The object key name (or key) that uniquely identifies the object in the bucket. For more information, see Naming Amazon S3 objects.
`sequence_number`	Yes	String	The sequence number, which is an ordinal that's included in the records for a given object. To order records of the same bucket and key, you can sort on `sequence_number`. For a given bucket and key, a lexicographically larger `sequence_number` value implies that the record was introduced to the bucket more recently.
`record_type`	Yes	String	The type of this record, one of `CREATE`, `UPDATE_METADATA`, or `DELETE`. `CREATE` records indicate that a new object (or a new version of the object) was written to the bucket. `UPDATE_METADATA` records capture changes to mutable metadata for an existing object, such as the storage class or tags. `DELETE` records indicate that this object (or this version of the object) has been deleted. When versioning is enabled, `DELETE` records represent either a delete marker or a permanent delete. Delete markers have a `record_type` value of `DELETE` and an `is_delete_marker` value of `True`. Permanent delete records have null values in all other columns except `bucket`, `key`, `sequence_number`, `record_type`, `record_timestamp`, and `version_id`. For more information, see Deleting object versions from a versioning-enabled bucket.
`record_timestamp`	Yes	Timestamp NTZ (no time zone)	The timestamp that's associated with this record.
`version_id`	No	String	The object's version ID. When you enable versioning on a bucket, Amazon S3 assigns a version number to objects that are added to the bucket. For more information, see Retaining multiple versions of objects with S3 Versioning. Objects that are stored in your bucket before you set the versioning state have a version ID of null.
`is_delete_marker`	No	Boolean	The object's delete marker status. If the object is a delete marker, this value is `True`. Otherwise, it's `False`. For more information, see Working with delete markers. Note Rows that are added for delete markers have a `record_type` value of `DELETE`, not `UPDATE_METADATA`. If the delete marker is created as the result of an S3 Lifecycle expiration, the `requester` value is `s3.amazonaws.com`.
`size`	No	Long	The object size in bytes, not including the size of incomplete multipart uploads or object metadata. If `is_delete_marker` is `True`, the size is `0`. For more information, see System-defined object metadata.
`last_modified_date`	No	Timestamp NTZ (no time zone)	The object creation date or the last modified date, whichever is the latest. For multipart uploads, the object creation date is the date when the multipart upload is initiated. For more information, see System-defined object metadata.
`e_tag`	No	String	The entity tag (ETag), which is a hash of the object. The ETag reflects changes only to the contents of an object, not to its metadata. The ETag can be an MD5 digest of the object data. Whether the ETag is an MD5 digest depends on how the object was created and how it's encrypted. For more information, see Object in the Amazon S3 API Reference.
`storage_class`	No	String	The storage class that’s used for storing the object. One of `STANDARD`, `REDUCED_REDUNDANCY`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `DEEP_ARCHIVE`, or `GLACIER_IR`. For more information, see Understanding and managing Amazon S3 storage classes.
`is_multipart`	No	Boolean	The object's upload type. If the object was uploaded as a multipart upload, this value is `True`. Otherwise, it's `False`. For more information, see Uploading and copying objects using multipart upload in Amazon S3.
`encryption_status`	No	String	The object's server-side encryption status, depending on what kind of encryption key is used: server-side encryption with Amazon S3 managed keys (SSE-S3), server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS), dual-layer server-side encryption with AWS KMS keys (DSSE-KMS), or server-side encryption with customer-provided keys (SSE-C). If the object is unencrypted, this value is null. Possible values are `SSE-S3`, `SSE-KMS`, `DSSE-KMS`, `SSE-C`, or null. For more information, see Protecting data with encryption.
`is_bucket_key_enabled`	No	Boolean	The object's S3 Bucket Key enablement status. If the object uses an S3 Bucket Key for SSE-KMS, this value is `True`. Otherwise, it's `False`. For more information, see Configuring an S3 Bucket Key at the object level.
`kms_key_arn`	No	String	The Amazon Resource Name (ARN) for the KMS key with which the object is encrypted, for rows where `encryption_status` is `SSE-KMS` or `DSSE-KMS`. If the object isn't encrypted with SSE-KMS or DSSE-KMS, the value is null. For more information, see Using server-side encryption with AWS KMS keys (SSE-KMS) and Using dual-layer server-side encryption with AWS KMS keys (DSSE-KMS). Note If a row represents an object version that no longer existed at the time that a delete or overwrite event was processed, `kms_key_arn` contains a null value, even if the `encryption_status` column value is `SSE-KMS` or `DSSE-KMS`.
`checksum_algorithm`	No	String	The algorithm that’s used to create the checksum for the object, one of `CRC64-NVME`, `CRC32`, `CRC32C`, `SHA1`, or `SHA256`. If no checksum is present, this value is null. For more information, see Using supported checksum algorithms.
`object_tags`	No	Map <String, String>	The object tags that are associated with the object. Object tags are stored as a map of key-value pairs. If an object has no object tags, an empty map (`{}`) is stored. For more information, see Categorizing your storage using tags Note If the `record_type` value is `DELETE`, the `object_tags` column contains a null value. If the `record_type` value is `CREATE` or `UPDATE_METADATA`, rows that represent object versions that no longer existed at the time that a delete or overwrite event was processed will contain a null value in the `object_tags` column.
`user_metadata`	No	Map <String, String>	The user metadata that's associated with the object. User metadata is stored as a map of key-value pairs. If an object has no user metadata, an empty map (`{}`) is stored. For more information, see User-defined object metadata. Note If the `record_type` value is `DELETE`, the `user_metadata` column contains a null value. If the `record_type` value is `CREATE` or `UPDATE_METADATA`, rows that represent object versions that no longer existed at the time that a delete or overwrite event was processed will contain a null value in the `user_metadata` column.
`requester`	No	String	The AWS account ID of the requester or the AWS service principal that made the request.
`source_ip_address`	No	String	The source IP address of the request. For records that are generated by a user request, this column contains the source IP address of the request. For actions taken by Amazon S3 or another AWS service on behalf of the user, this column contains a null value.
`request_id`	No	String	The request ID that's associated with the request.

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Limitations and restrictions

Configuring metadata tables