# Working with objects in Amazon S3 To store your data in Amazon S3, you work with resources known as buckets and objects. A *bucket* is a container for objects. An *object* is a file and any metadata that describes that file. To store an object in Amazon S3, you create a bucket and then upload the object to a bucket. When the object is in the bucket, you can open it, download it, and copy it. When you no longer need an object or a bucket, you can clean up these resources. **Note** For more information about using the Amazon S3 Express One Zone storage class with directory buckets, see [S3 Express One Zone](directory-bucket-high-performance.md#s3-express-one-zone) and [Working with directory buckets](directory-buckets-overview.md). **Important** In the Amazon S3 console, when you choose **Open** or **Download As** for an object, these operations create presigned URLs. For the duration of five minutes, your object will be accessible to anyone who has access to these presigned URLs. For more information about presigned URLs, see [Using presigned URLS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html). With Amazon S3, you pay only for what you use. For more information about Amazon S3 features and pricing, see [Amazon S3](https://aws.amazon.com/s3). If you are a new Amazon S3 customer, you can get started with Amazon S3 for free. For more information, see [AWS Free Tier](https://aws.amazon.com/free). **Topics** + [ # Amazon S3 objects overview ](UsingObjects.md) + [ # Naming Amazon S3 objects ](object-keys.md) + [ # Working with object metadata ](UsingMetadata.md) + [ # Uploading objects ](upload-objects.md) + [ # Add preconditions to S3 operations with conditional requests ](conditional-requests.md) + [ # Copying, moving, and renaming objects ](copy-object.md) + [ # Downloading objects ](download-objects.md) + [ # Checking object integrity in Amazon S3 ](checking-object-integrity.md) + [ # Deleting Amazon S3 objects ](DeletingObjects.md) + [ # Organizing, listing, and working with your objects ](organizing-objects.md) + [ # Download and upload objects with presigned URLs ](using-presigned-url.md) + [ # Transforming objects with S3 Object Lambda ](transforming-objects.md) + [ # Performing object operations in bulk with Batch Operations ](batch-ops.md) + [ # Querying data in place with Amazon S3 Select ](selecting-content-from-objects.md) # Amazon S3 objects overview Amazon S3 is an object store that uses unique key-values to store as many objects as you want. You store these objects in one or more buckets, and each object can be up to 50 TB in size. An object consists of the following: Key The name that you assign to an object. You use the object key to retrieve the object. For more information, see [Working with object metadata](UsingMetadata.md). Version ID Within a bucket, a key and version ID uniquely identify an object. The version ID is a string that Amazon S3 generates when you add an object to a bucket. For more information, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). Value The content that you are storing. An object value can be any sequence of bytes. Objects can range in size from zero to 50 TB. For more information, see [Uploading objects](upload-objects.md). While Amazon S3 documentation commonly references 50 TB object size limit, the actual maximum object size is 53.7 TB (48.8 TiB). This limit is determined by multipart upload constraints: 10,000 maximum parts x 5 GiB per part = 50,000 GiB (53.7 TB) Metadata A set of name-value pairs with which you can store information regarding the object. You can assign metadata, referred to as user-defined metadata, to your objects in Amazon S3. Amazon S3 also assigns system-metadata to these objects, which it uses for managing objects. For more information, see [Working with object metadata](UsingMetadata.md). Subresources Amazon S3 uses the subresource mechanism to store object-specific additional information. Because subresources are subordinates to objects, they are always associated with some other entity such as an object or a bucket. For more information, see [Object subresources](#ObjectAndSubResource). Access control information You can control access to the objects you store in Amazon S3. Amazon S3 supports both the resource-based access control, such as an access control list (ACL) and bucket policies, and user-based access control. For more information about access control, see the following: + [Access control in Amazon S3](access-management.md) + [Identity and Access Management for Amazon S3](security-iam.md) + [Configuring ACLs](managing-acls.md) Your Amazon S3 resources (for example, buckets and objects) are private by default. You must explicitly grant permission for others to access these resources. For more information about sharing objects, see [Sharing objects with presigned URLs](ShareObjectPreSignedURL.md). Tags You can use tags to categorize your stored objects, for access control, or cost allocation. For more information, see [Categorizing your objects using tags](object-tagging.md). ## Object subresources Amazon S3 defines a set of subresources associated with buckets and objects. Subresources are subordinates to objects. This means that subresources don't exist on their own. They are always associated with some other entity, such as an object or a bucket. The following table lists the subresources associated with Amazon S3 objects. | Subresource | Description | | --- | --- | | acl | Contains a list of grants identifying the grantees and the permissions granted. When you create an object, the acl identifies the object owner as having full control over the object. You can retrieve an object ACL or replace it with an updated list of grants. Any update to an ACL requires you to replace the existing ACL. For more information about ACLs, see [Access control list (ACL) overview](acl-overview.md). | # Naming Amazon S3 objects The *object key* (or key name) uniquely identifies the object in an Amazon S3 bucket. When you create an object, you specify the key name. For example, on the [Amazon S3 console](https://console.aws.amazon.com/s3/home), when you select a bucket, a list of objects in your bucket appears. These names are the *object keys*. The object key name consists of a sequence of Unicode characters encoded in UTF-8, with a maximum length of 1,024 bytes or approximately 1,024 Latin characters. In some locales, a single character may require 2 bytes for encoding. When naming your objects, be aware of the following: + Object key names are case sensitive. + Object key names include any prefixes (known as *folders* in the console). For example, `Development/Projects.xls` is the full object key name of the `Projects.xls` object located within the `Development` prefix (or folder). The prefix, the delimiter (`/`), and the name of the object are included in the 1,024 byte limitation for the object key name. For more information about prefixes and folders, see [Choosing object key names](#object-key-choose). + Certain characters might require special handling when they're used in object key names. For more information, see [Object key naming guidelines](#object-key-guidelines). **Note** Object key names with the value `"soap"` aren't supported for [virtual-hosted-style requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access). For object key name values where `"soap"` is used, a [path-style URL](https://docs.aws.amazon.com//AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) must be used instead. ## Choosing object key names The Amazon S3 data model is a flat structure: You create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders. However, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports a concept of folders. For more information about how to edit metadata from the Amazon S3 console, see [Editing object metadata in the Amazon S3 console](add-object-metadata.md). Suppose that your bucket (`admin-created`) has four objects with the following object keys: `Development/Projects.xls` `Finance/statement1.pdf` `Private/taxdocument.pdf` `s3-dg.pdf` The console uses the key name prefixes (`Development/`, `Finance/`, and `Private/`) and delimiter (`/`) to present a folder structure. The `s3-dg.pdf` key doesn't contain a slash-delimited prefix, so its object appears directly at the root level of the bucket. If you open the `Development/` folder, you see the `Projects.xlsx` object in it. + Amazon S3 supports buckets and objects, and there is no hierarchy. However, by using prefixes and delimiters in an object key name, the Amazon S3 console and the AWS SDKs can infer hierarchy and introduce the concept of folders. + The Amazon S3 console implements folder object creation by creating a zero-byte object with the folder *prefix and delimiter* value as the key. These folder objects don't appear in the console. Otherwise they behave like any other objects and can be viewed and manipulated through the REST API, AWS CLI, and AWS SDKs. ## Object key naming guidelines You can use any UTF-8 character in an object key name. However, using certain characters in key names can cause problems with some applications and protocols. The following guidelines help you maximize compliance with DNS, web-safe characters, XML parsers, and other APIs. ### Safe characters The following character sets are generally safe for use in key names: | | | | --- |--- | | Alphanumeric characters | 0-9 a-z A-Z | | Special characters | Exclamation point (`!`) Hyphen (`-`) Underscore (`_`) Period (`.`) Asterisk (`*`) Single quotation mark (`'`) Opening parenthesis (`(`) Closing parenthesis (`)`) | The following are examples of valid object key names: + `4my-organization` + `my.great_photos-2014/jan/myvacation.jpg` + `videos/2014/birthday/video1.wmv` **Note** If you use the Amazon S3 console to download objects that have key names that end with periods (`.`), the periods are removed from the ends of the key names of the downloaded objects. To retain periods at the ends of key names in downloaded objects, you must use the AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. In addition, be aware of the following prefix limitations: Objects with a prefix of `./` must be uploaded or downloaded with the AWS CLI, AWS SDKs, or REST API. You can't use the Amazon S3 console to upload these objects. Object keys that contain relative path elements (for example, `../`) are valid if, when parsed left-to-right, the cumulative count of relative path segments never exceeds the number of non-relative path elements encountered. This rule applies to all requests made by using the Amazon S3 console, Amazon S3 REST API, AWS CLI, and AWS SDKs. For example: `videos/2014/../../video1.wmv` is valid. `videos/../../video1.wmv` isn't valid. `videos/../../2014/video1.wmv` isn't valid. ### Period-only path segments Object keys containing period-only path segments (`.` or `..`) can cause unexpected behavior when processed by applications, SDKs, or tools that interpret these as relative path references. The following patterns can cause issues: + `folder/./file.txt` - Contains current directory reference + `folder/../file.txt` - Contains parent directory reference + `./file.txt` - Starts with current directory reference + `../file.txt` - Starts with parent directory reference The following patterns work normally: + `folder/.hidden/file.txt` - Period is part of filename, not standalone + `folder/..backup/file.txt` - Periods are part of filename, not standalone When applications process object keys with period-only segments, the following behavioral impacts can occur: + *Path normalization* - Many systems automatically resolve `.` and `..` references, potentially changing the effective path (for example, `folder/./file.txt` becomes `folder/file.txt`) + *Access issues* - Applications might fail to locate objects due to path resolution differences + *Inconsistent behavior* - Different tools and SDKs might handle these patterns differently **Important** To avoid these issues, we recommend avoiding period-only path segments in object key names. Use alternative naming conventions for organizational purposes. ### Characters that might require special handling The following characters in a key name might require additional code handling and most likely must be URL encoded or referenced as HEX. Some of these characters are non-printable characters that your browser might not handle, which also require special handling: + Ampersand (`&`) + Dollar sign (`$`) + ASCII character ranges 00–1F hex (0–31 decimal) and 7F (127 decimal) + At symbol (`@`) + Equal sign (`=`) + Semicolon (`;`) + Forward slash (`/`) + Colon (`:`) + Plus sign (`+`) + Space – Significant sequences of spaces might be lost in some cases (especially multiple spaces) + Comma (`,`) + Question mark (`?`) ### Characters to avoid We recommend not using the following characters in a key name because of significant special character handling, which isn't consistent across all applications: + Backslash (`\`) + Left brace (`{`) + Non-printable ASCII characters (128–255 decimal characters) + Caret or circumflex (`^`) + Right brace (`}`) + Percent character (`%`) + Grave accent or backtick (```) + Right bracket (`]`) + Quotation mark (`"`) + Greater than sign (`>`) + Left bracket (`[`) + Tilde (`~`) + Less than sign (`<`) + Pound sign (`#`) + Vertical bar or pipe (`|`) ### XML-related object key constraints As specified by the [XML standard on end-of-line handling](https://www.w3.org/TR/REC-xml/#sec-line-ends), all XML text is normalized such that single carriage returns (ASCII code 13) and carriage returns immediately followed by a line feed (ASCII code 10), also known as newline characters, are replaced by a single line feed character. To ensure the correct parsing of object keys in XML requests, carriage returns and [other special characters must be replaced with their equivalent XML entity code](https://www.w3.org/TR/xml/#syntax) when they're inserted within XML tags. The following is a list of such special characters and their equivalent XML entity codes: + Apostrophe (`'`) must be replaced with `'` + Quotation mark (`"`) must be replaced with `"` + Ampersand (`&`) must be replaced with `&` + Less than sign (`<`) must be replaced with `<` + Greater than sign (`>`) must be replaced with `>` + Carriage return (`\r`) must be replaced with ` ` or ` ` + Newline (`\n`) must be replaced with ` ` or ` ` **Example** The following example illustrates the use of an XML entity code as a substitution for a carriage return. This `DeleteObjects` request deletes an object with the `key` parameter `/some/prefix/objectwith\rcarriagereturn` (where the `\r` is the carriage return). ``` /some/prefix/objectwith carriagereturn ``` ## Object key sort order Amazon S3 sorts object keys, including prefixes, lexicographically by their UTF-8 encoded byte values. ASCII characters sort in the following order: + Special characters (for example, `!`, `/`) + Uppercase letters (A–Z) + Lowercase letters (a–z) Non-ASCII characters (for example, é, 中 文) are encoded as multi-byte UTF-8 sequences and typically sort after ASCII characters because of their higher byte values (for example, `0xC3` for é, `0xE4` for 中). For example, prefixes such as `apple/`, `Apple/`, `éclair/`, `中 文/` would sort as: 1. `Apple/` (starts with `0x41`) 2. `apple/` (starts with `0x61`) 3. `éclair/` (starts with `0xC3 0xA9`) 4. `中 文/` (starts with `0xE4 0xB8 0xAD` `0xE6 0x96 0x87`) # Working with object metadata There are two kinds of object metadata in Amazon S3: *system-defined metadata* and *user-defined metadata*. System-defined metadata includes metadata such as the object's creation date, size, and storage class. User-defined metadata is metadata that you can choose to set at the time that you upload an object. This user-defined metadata is a set of name-value pairs. For more information, see [System-defined object metadata](#SysMetadata) and [User-defined object metadata](#UserMetadata). When you create an object, you specify the *object key* (or *key name*), which uniquely identifies the object in an Amazon S3 bucket. For more information, see [Naming Amazon S3 objects](object-keys.md). You can also set [user-defined metadata](#UserMetadata) in Amazon S3 at the time that you upload the object. After you upload the object, you can't modify this user-defined metadata. The only way to modify this metadata is to make a copy of the object and set the metadata. For more information about editing metadata by using the Amazon S3 console, see [Editing object metadata in the Amazon S3 console](add-object-metadata.md). **Query your metadata and accelerate data discovery with S3 Metadata** To easily find, store, and query metadata for your S3 objects, you can use S3 Metadata. With S3 Metadata, you can quickly prepare data for use in business analytics, content retrieval, artificial intelligence and machine learning (AI/ML) model training, and more. S3 Metadata accelerates data discovery by automatically capturing metadata for the objects in your general purpose buckets and storing it in read-only, fully managed Apache Iceberg tables that you can query. These read-only tables are called *metadata tables*. As objects are added to, updated, and removed from your general purpose buckets, S3 Metadata automatically refreshes the corresponding metadata tables to reflect the latest changes. By default, S3 Metadata provides [system-defined object metadata](#SysMetadata), such as an object's creation time and storage class, and custom metadata, such as tags and [user-defined metadata](#UserMetadata) that was included during object upload. S3 Metadata also provides event metadata, such as when an object is updated or deleted, and the AWS account that made the request. Metadata tables are stored in S3 table buckets, which provide storage that's optimized for tabular data. To query your metadata, you can integrate your table bucket with AWS analytics services, such as Amazon Athena, Amazon Redshift, and Amazon Quick. For more information about S3 Metadata, see [Discovering your data with S3 Metadata tables](metadata-tables-overview.md). ## System-defined object metadata For each object stored in a bucket, Amazon S3 maintains a set of system metadata. Amazon S3 processes this system metadata as needed. For example, Amazon S3 maintains object-creation date and size metadata, using this information as part of object management. There are two categories of system metadata: + **System controlled** – Metadata such as the object-creation date is system controlled, which means that only Amazon S3 can modify the date value. + **User controlled** – Other system metadata, such as the storage class configured for the object and whether the object has server-side encryption enabled, are examples of system metadata whose values you control. If your bucket is configured as a website, sometimes you might want to redirect a page request either to another page or an external URL. In this case, a webpage is an object in your bucket. Amazon S3 stores the page redirect value as system metadata, which you can control. When you create objects, you can configure the values of these system metadata items or update the values when you need to. For more information about storage classes, see [Understanding and managing Amazon S3 storage classes](storage-class-intro.md). Amazon S3 uses AWS KMS keys to encrypt your Amazon S3 objects. AWS KMS encrypts only the object data. The checksum and the specified algorithm are stored as part of the object's metadata. If server-side encryption is requested for the object, then the checksum is stored in encrypted form. For more information about server-side encryption, see [Protecting data with encryption](UsingEncryption.md). **Note** The `PUT` request header is limited to 8 KB in size. Within the `PUT` request header, the system-defined metadata is limited to 2 KB in size. The size of system-defined metadata is measured by taking the sum of the number of bytes in the US-ASCII encoding of each key and value. The following table provides a list of system-defined metadata and whether you can update it. | Name | Description | Can user modify the value? | | --- | --- | --- | | Date | The current date and time. | No | | Cache-Control | A general header field used to specify caching policies. | Yes | | Content-Disposition | Object presentational information. | Yes | | Content-Encoding | The content encodings (like compression) that have been applied to the object's data | Yes | | Content-Length | The object size in bytes. | No | | Content-Type | The object type. | Yes | | Last-Modified | The object creation date or the last modified date, whichever is the latest. For multipart uploads, the object creation date is the date of initiation of the multipart upload. | No | | ETag | An entity tag (ETag) that represents a specific version of an object. For objects that are not uploaded as a multipart upload and are either unencrypted or encrypted by server-side encryption with Amazon S3 managed keys (SSE-S3), the ETag is an MD5 digest of the data. | No | | x-amz-server-side-encryption | A header that indicates whether server-side encryption is enabled for the object, and whether that encryption is using the AWS Key Management Service (AWS KMS) keys (SSE-KMS) or using Amazon S3 managed encryption keys (SSE-S3). For more information, see [Protecting data with server-side encryption](serv-side-encryption.md). | Yes | | x-amz-checksum-crc64nvme, x-amz-checksum-crc32, x-amz-checksum-crc32c, x-amz-checksum-sha1, x-amz-checksum-sha256 | Headers that contain the checksum or digest of the object. At most, one of these headers will be set at a time, depending on the checksum algorithm that you instruct Amazon S3 to use. For more information about choosing the checksum algorithm, see [Checking object integrity in Amazon S3](checking-object-integrity.md). | No | | x-amz-checksum-type | The checksum type, which determines how part-level checksums are combined to create an object-level checksum for multipart objects. | Yes | | x-amz-version-id | The object version. When you enable versioning on a bucket, Amazon S3 assigns a version ID to objects added to the bucket. For more information, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). | No | | x-amz-delete-marker | A Boolean marker that indicates whether the object is a delete marker. This marker is used only in buckets that have versioning enabled, | No | | x-amz-storage-class | The storage class used for storing the object. For more information, see [Understanding and managing Amazon S3 storage classes](storage-class-intro.md). | Yes | | x-amz-website-redirect-location | A header that redirects requests for the associated object to another object in the same bucket or to an external URL. For more information, see [(Optional) Configuring a webpage redirect](how-to-page-redirect.md). | Yes | | x-amz-server-side-encryption-aws-kms-key-id | A header that indicates the ID of the AWS KMS symmetric encryption KMS key that was used to encrypt the object. This header is used only when the x-amz-server-side-encryption header is present and has the value of aws:kms. | Yes | | x-amz-server-side-encryption-customer-algorithm | A header that indicates whether server-side encryption with customer-provided encryption keys (SSE-C) is enabled. For more information, see [Using server-side encryption with customer-provided keys (SSE-C)](ServerSideEncryptionCustomerKeys.md). | Yes | | x-amz-tagging | The tag-set for the object. The tag-set must be encoded as URL Query parameters. | Yes | ## User-defined object metadata When uploading an object, you can also assign metadata to the object. You provide this optional information as a name-value (key-value) pair when you send a `PUT` or `POST` request to create the object. When you upload objects using the REST API, the optional user-defined metadata names must begin with `x-amz-meta-` to distinguish them from other HTTP headers. When you retrieve the object using the REST API, this prefix is returned. When you upload objects using the SOAP API, the prefix is not required. When you retrieve the object using the SOAP API, the prefix is removed, regardless of which API you used to upload the object. **Note** SOAP APIs for Amazon S3 are not available for new customers, and are approaching End of Life (EOL) on August 31, 2025. We recommend that you use either the REST API or the AWS SDKs. When metadata is retrieved through the REST API, Amazon S3 combines headers that have the same name (ignoring case) into a comma-delimited list. If some metadata contains unprintable characters, it is not returned. Instead, the `x-amz-missing-meta` header is returned with a value of the number of unprintable metadata entries. The `HeadObject` action retrieves metadata from an object without returning the object itself. This operation is useful if you're only interested in an object's metadata. To use `HEAD`, you must have `READ` access to the object. For more information, see [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) in the * Amazon Simple Storage Service API Reference*. User-defined metadata is a set of key-value pairs. Amazon S3 stores user-defined metadata keys in lowercase. Amazon S3 allows arbitrary Unicode characters in your metadata values. To avoid issues related to the presentation of these metadata values, you should conform to using US-ASCII characters when using REST and UTF-8 when using SOAP or browser-based uploads through `POST`. When using non-US-ASCII characters in your metadata values, the provided Unicode string is examined for non-US-ASCII characters. Values of such headers are character decoded as per [RFC 2047](https://datatracker.ietf.org/doc/html/rfc2047) before storing and encoded as per [RFC 2047](https://datatracker.ietf.org/doc/html/rfc2047) to make them mail-safe before returning. If the string contains only US-ASCII characters, it is presented as is. The following is an example. ``` PUT /Key HTTP/1.1 Host: amzn-s3-demo-bucket.s3.amazonaws.com x-amz-meta-nonascii: ÄMÄZÕÑ S3 HEAD /Key HTTP/1.1 Host: amzn-s3-demo-bucket.s3.amazonaws.com x-amz-meta-nonascii: =?UTF-8?B?w4PChE3Dg8KEWsODwpXDg8KRIFMz?= PUT /Key HTTP/1.1 Host: amzn-s3-demo-bucket.s3.amazonaws.com x-amz-meta-ascii: AMAZONS3 HEAD /Key HTTP/1.1 Host: amzn-s3-demo-bucket.s3.amazonaws.com x-amz-meta-ascii: AMAZONS3 ``` **Note** The `PUT` request header is limited to 8 KB in size. Within the `PUT` request header, the user-defined metadata is limited to 2 KB in size. The size of user-defined metadata is measured by taking the sum of the number of bytes in the UTF-8 encoding of each key and value. For information about changing the metadata of your object after it has been uploaded by creating a copy of the object, modifying it, and replacing the old object, or creating a new version, see [Editing object metadata in the Amazon S3 console](add-object-metadata.md). # Editing object metadata in the Amazon S3 console You can use the Amazon S3 console to edit metadata for existing S3 objects by using the **Copy** action. To edit metadata, you copy objects to the same destination and specify the new metadata you want to apply, which replaces the old metadata for the object. Some metadata is set by Amazon S3 when you upload the object. For example, `Content-Length` and `Last-Modified` are system-defined object metadata fields that can't be modified by a user. You can also set user-defined metadata when you upload the object and replace it as your needs change. For example, you might have a set of objects that you initially store in the `STANDARD` storage class. Over time, you may no longer need this data to be highly available. So, you can change the storage class to `GLACIER` by replacing the value of the `x-amz-storage-class` key from `STANDARD` to `GLACIER`. **Note** Consider the following when you are replacing object metadata in Amazon S3: You must specify existing metadata you want to retain, metadata you want to add, and metadata you want to edit. If your object is less than 5 GB, you can use the **Copy** action in the S3 console to replace object metadata. If your object is greater than 5 GB, you can replace the object metadata when you copy an object with multipart upload by using the [AWS CLI](mpu-upload-object.md#UsingCLImpUpload) or [AWS SDKs](CopyingObjectsMPUapi.md). For more information, see [Copying an object using multipart upload](CopyingObjectsMPUapi.md). For a list of additional permissions required to replace metadata, see [Required permissions for Amazon S3 API operations](using-with-s3-policy-actions.md). For example policies that grant this permission, see [Identity-based policy examples for Amazon S3](example-policies-s3.md). This action creates a *copy* of the object with updated settings and the last-modified date. If S3 Versioning is enabled, a new version of the object is created, and the existing object becomes an older version. If S3 Versioning isn't enabled, a new copy of the object replaces the original object. The AWS account associated with the IAM role that changes the property also becomes the owner of the new object or (object version). Editing metadata replaces values for existing key names. Objects that are encrypted with customer-provided encryption keys (SSE-C) can't be copied by using the console. You must use the AWS CLI, AWS SDK, or the Amazon S3 REST API. When copying an object by using the Amazon S3 console, you might receive the error message "Copied metadata can't be verified." The console uses headers to retrieve and set metadata for your object. If your network or browser configuration modifies your network requests, this behavior might cause unintended metadata (such as modified `Cache-Control` headers) to be written to your copied object. Amazon S3 can't verify this unintended metadata. To address this issue, check your network and browser configuration to make sure it doesn't modify headers, such as `Cache-Control`. For more information, see [The Shared Responsibility Model](https://docs.aws.amazon.com/whitepapers/latest/applying-security-practices-to-network-workload-for-csps/the-shared-responsibility-model.html). **Warning** When replacing metadata for folders, wait for the **Copy** action to finish before adding new objects to the folder. Otherwise, new objects might also be edited. The following topics describe how to replace metadata for an object by using the **Copy** action in the Amazon S3 console. ## Replacing system-defined metadata You can replace some system-defined metadata for an S3 object. For a list of system-defined metadata and values that you can modify, see [System-defined object metadata](UsingMetadata.md#SysMetadata). **To replace system-defined metadata of an object** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets** or **Directory buckets**. 1. In the list of buckets, choose the name of the bucket that contains the objects you want to change. 1. Select the check box for the objects you want to change. 1. On the **Actions** menu, choose **Copy** from the list of options that appears. 1. To specify the destination path, choose **Browse S3**, navigate to the same destination as the source objects, and select the destination check box. Choose **Choose destination** in the lower-right corner. Alternatively, enter the destination path. 1. If you do *not* have bucket versioning enabled, you will see a warning recommending you enable Bucket Versioning to help protect against unintentionally overwriting or deleting objects. If you want to keep all versions of objects in this bucket, select **Enable Bucket Versioning**. You can also view the default encryption and Object Lock properties in **Destination details**. 1. Under **Additional copy settings**, choose **Specify settings** to specify settings for **Metadata**. 1. Scroll to the **Metadata** section, and then choose **Replace all metadata**. 1. Choose **Add metadata**. 1. For metadata **Type**, select **System-defined**. 1. Specify a unique **Key** and the metadata **Value**. 1. To edit additional metadata, choose **Add metadata**. You can also choose **Remove** to remove a set of type-key-values. 1. Choose **Copy**. Amazon S3 saves your metadata changes. ## Replacing user-defined metadata You can replace user-defined metadata of an object by combining the metadata prefix, `x-amz-meta-`, and a name you choose to create a custom key. For example, if you add the custom name `alt-name`, the metadata key would be `x-amz-meta-alt-name`. User-defined metadata can be as large as 2 KB total. To calculate the total size of user-defined metadata, sum the number of bytes in the UTF-8 encoding for each key and value. Both keys and their values must conform to US-ASCII standards. For more information, see [User-defined object metadata](UsingMetadata.md#UserMetadata). **To replace user-defined metadata of an object** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the navigation pane, choose **Buckets**, and then choose the **General purpose buckets** or **Directory buckets** tab. Navigate to the Amazon S3 bucket or folder that contains the objects you want to change. 1. Select the check box for the objects you want to change. 1. On the **Actions** menu, choose **Copy** from the list of options that appears. 1. To specify the destination path, choose **Browse S3**, navigate to the same destination as the source objects, and select the destination check box. Choose **Choose destination**. Alternatively, enter the destination path. 1. If you do *not* have bucket versioning enabled, you will see a warning recommending you enable Bucket Versioning to help protect against unintentionally overwriting or deleting objects. If you want to keep all versions of objects in this bucket, select **Enable Bucket Versioning**. You can also view the default encryption and Object Lock properties in **Destination details**. 1. Under **Additional copy settings**, choose **Specify settings** to specify settings for **Metadata**. 1. Scroll to the **Metadata** section, and then choose **Replace all metadata**. 1. Choose **Add metadata**. 1. For metadata **Type**, choose **User-defined**. 1. Enter a unique custom **Key** following `x-amz-meta-`. Also enter a metadata **Value**. 1. To add additional metadata, choose **Add metadata**. You can also choose **Remove** to remove a set of type-key-values. 1. Choose **Copy**. Amazon S3 saves your metadata changes. # Discovering your data with S3 Metadata tables Amazon S3 Metadata accelerates data discovery by automatically capturing metadata for objects in your general purpose buckets and storing it in read-only, fully managed Apache Iceberg tables that you can query. These read-only tables are called *metadata tables*. As objects are added to, updated, or removed from your general purpose buckets, S3 Metadata automatically refreshes the corresponding metadata tables to reflect the latest changes. By default, S3 Metadata provides three types of metadata: + [System-defined metadata](UsingMetadata.md#SysMetadata), such as an object's creation time and storage class + Custom metadata, such as tags and [user-defined metadata](UsingMetadata.md#UserMetadata) that was included during object upload + Event metadata, such as when an object is updated or deleted, and the AWS account that made the request With S3 Metadata, you can easily find, store, and query metadata for your S3 objects, so that you can quickly prepare data for use in business analytics, content retrieval, artificial intelligence and machine learning (AI/ML) model training, and more. For each general purpose bucket, you can create a metadata table configuration that contains two complementary metadata tables: + **Journal table** – By default, your metadata table configuration contains a *journal table*, which captures events that occur for the objects in your bucket. The journal table records changes made to your data in near real time, helping you to identify new data uploaded to your bucket, track recently deleted objects, monitor lifecycle transitions, and more. The journal table records new objects and updates to your objects and their metadata (those updates that require either a `PUT` or a `DELETE` operation). The journal table captures metadata only for change events (such as uploads, updates, and deletes) that happen after you create your metadata table configuration. Because this table is queryable, you can audit the changes to your bucket through simple SQL queries. The journal table is required for each metadata table configuration. (In the initial release of S3 Metadata, the journal table was referred to as "the metadata table.") For more information about what data is stored in journal tables, see [S3 Metadata journal tables schema](metadata-tables-schema.md). To help minimize your storage costs, you can choose to enable journal table record expiration. For more information, see [Expiring journal table records](metadata-tables-expire-journal-table-records.md). + **Live inventory table** – Optionally, you can add a *live inventory table* to your metadata table configuration. The live inventory table provides a simple, queryable inventory of all the objects and their versions in your bucket so that you can determine the latest state of your data. You can use the live inventory table to simplify and speed up business workflows and big data jobs by identifying objects that you want to process for various workloads. For example, you can query the live inventory table to find all objects stored in a particular storage class, all objects with certain tags, all objects that aren't encrypted with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), and more. When you enable the live inventory table for your metadata table configuration, the table goes through a process known as *backfilling*, during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects that exist in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your live inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the live inventory table within one hour. You're charged for backfilling your inventory table. If your general purpose bucket has more than one billion objects, you're also charged a monthly fee for your live inventory table. For more information, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). For more information about what data is stored in live inventory tables, see [S3 Metadata live inventory tables schema](metadata-tables-inventory-schema.md). Your metadata tables are stored in an AWS managed S3 table bucket, which provides storage that's optimized for tabular data. To query your metadata, you can integrate your table bucket with Amazon SageMaker Lakehouse. This integration, which uses the AWS Glue Data Catalog and AWS Lake Formation, allows AWS analytics services to automatically discover and access your table data. After your table bucket is integrated with the AWS Glue Data Catalog, you can directly query your metadata tables with AWS analytics services such as Amazon Athena, Amazon EMR, and Amazon Redshift. You can also create interactive dashboards with your query data by using Amazon Quick. For more information about integrating your AWS managed S3 table bucket with Amazon SageMaker Lakehouse, see [Integrating Amazon S3 Tables with AWS analytics services](s3-tables-integrating-aws.md). You can also query your metadata tables with Apache Spark, Apache Trino, and any other application that supports the Apache Iceberg format by using the AWS Glue Iceberg REST endpoint, the Amazon S3 Tables Iceberg REST endpoint, or the Amazon S3 Tables Catalog for Apache Iceberg client catalog. For more information about accessing your metadata tables, see [Accessing table data](s3-tables-access.md). For S3 Metadata pricing, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). ## How metadata tables work Metadata tables are managed by Amazon S3, and can't be modified by any IAM principal outside of Amazon S3 itself. You can, however, delete your metadata tables. As a result, metadata tables are read-only, which helps ensure that they correctly reflect the contents of your general purpose bucket. To generate and store object metadata in AWS managed metadata tables, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata tables to reflect the latest changes to your data as long as the configuration is active on the general purpose bucket. Before you create a metadata table configuration, make sure that you have the necessary AWS Identity and Access Management (IAM) permissions to create and manage metadata tables. For more information, see [Setting up permissions for configuring metadata tables](metadata-tables-permissions.md). **Metadata table storage, organization, and encryption** When you create your metadata table configuration, your metadata tables are stored in an AWS managed table bucket. All metadata table configurations in your account and in the same Region are stored in a single AWS managed table bucket. These AWS managed table buckets are named `aws-s3` and have the following Amazon Resource Name (ARN) format: `arn:aws:s3tables:region:account_id:bucket/aws-s3` For example, if your account ID is 123456789012 and your general purpose bucket is in US East (N. Virginia) (`us-east-1`), your AWS managed table bucket is also created in US East (N. Virginia) (`us-east-1`) and has the following ARN: `arn:aws:s3tables:us-east-1:123456789012:bucket/aws-s3` By default, AWS managed table buckets are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). After you create your first metadata configuration, you can set the default encryption setting for the AWS managed table bucket to use server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). For more information, see [Encryption for AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html#aws-managed-buckets-encryption) and [Specifying server-side encryption with AWS KMS keys (SSE-KMS) in table buckets](s3-tables-kms-specify.md). Within your AWS managed table bucket, the metadata tables for your configuration are typically stored in a namespace with the following naming format: `b_general-purpose-bucket-name` **Note** If your general purpose bucket name contains any periods, the periods are converted to underscores (`_`) in the namespace name. If your general purpose bucket was created before March 1, 2018, its name might contain uppercase letters and underscores, and it might also be up to 255 characters long. If your bucket name has these characteristics, your metadata table namespace will have a different format. The general purpose bucket name will be prefixed with `b_`, truncated to 63 characters, converted to all lowercase, and suffixed with a hash. Metadata tables have the following Amazon Resource Name (ARN) format, which includes the table ID of the metadata table: `arn:aws:s3tables:region-code:account-id:bucket/aws-s3/table/table-id` For example, a metadata table in the US East (N. Virginia) Region would have an ARN like the following: `arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3/table/a12bc345-67d8-912e-3456-7f89123g4h56` Journal tables have the name `journal`, and live inventory tables have the name `inventory`. When you create your metadata table configuration, you can choose to encrypt your AWS managed metadata tables with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. You can set the encryption type for your tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. To specify SSE-KMS for your metadata tables, you must have certain permissions. For more information, see [ Permissions for SSE-KMS](metadata-tables-permissions.md#metadata-kms-permissions). The encryption setting for a metadata table takes precedence over the default bucket-level encryption setting. If you don't specify encryption for a table, it will inherit the default encryption setting from the bucket. AWS managed table buckets don't count toward your S3 Tables quotas. For more information about working with AWS managed table buckets and AWS managed tables, see [Working with AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html). To monitor updates to your metadata table configuration, you can use AWS CloudTrail. For more information, see [Amazon S3 bucket-level actions that are tracked by CloudTrail logging](cloudtrail-logging-s3-info.md#cloudtrail-bucket-level-tracking). **Metadata table maintenance and record expiration** To keep your metadata tables performing at their best, Amazon S3 performs periodic maintenance activities on your tables, such as compaction and unreferenced file removal. These maintenance activities help to both minimize the cost of storing your metadata tables and optimize query performance. This table maintenance happens automatically, requiring no opt-in or ongoing management by you. **Note** You can't control the expiration of journal table or inventory table snapshots. For each table, Amazon S3 stores a minimum of 1 snapshot for a maximum of 24 hours. To help minimize your costs, you can configure journal table record expiration. By default, journal table records don't expire, and journal table records must be retained for a minimum of 7 days. For more information, see [Expiring journal table records](metadata-tables-expire-journal-table-records.md). **Topics** + [ ## How metadata tables work ](#metadata-tables-how-they-work) + [ # Metadata table limitations and restrictions ](metadata-tables-restrictions.md) + [ # S3 Metadata journal tables schema ](metadata-tables-schema.md) + [ # S3 Metadata live inventory tables schema ](metadata-tables-inventory-schema.md) + [ # Configuring metadata tables ](metadata-tables-configuring.md) + [ # Querying metadata tables ](metadata-tables-querying.md) + [ # Troubleshooting S3 Metadata ](metadata-tables-troubleshooting.md) # Metadata table limitations and restrictions Amazon S3 Metadata has the following limitations and restrictions: + S3 Metadata is currently available only in certain Regions. For more information, see [S3 Metadata AWS Regions](#metadata-tables-regions). + S3 Metadata supports all storage classes supported by general purpose buckets. For the S3 Intelligent-Tiering storage class, the specific tier isn't shown in the metadata table. + When you create a metadata table configuration, your metadata tables are stored in an AWS managed table bucket. You can't store your configuration in a customer-managed table bucket. + S3 Metadata isn't supported for directory buckets, table buckets, or vector buckets. You can create metadata table configurations only for general purpose buckets. The journal table captures metadata only for change events (such as uploads, updates, and deletes) that happen after you have created your metadata table configuration. + You can't control the expiration of journal table or inventory table snapshots. For each table, Amazon S3 stores a minimum of 1 snapshot for a maximum of 24 hours. To help minimize your costs, you can configure journal table record expiration. By default, journal table records don't expire, and journal table records must be retained for a minimum of 7 days. For more information, see [Expiring journal table records](metadata-tables-expire-journal-table-records.md). + You can create a metadata table configuration only for an entire general purpose bucket. You can't apply a metadata table configuration at the prefix level. + You can't pause and resume updates to metadata tables. However, you can delete your associated metadata configuration for journal or live inventory tables. Deleting your configuration doesn't delete the associated journal or inventory table. To re-create your configuration, you must first delete the old journal or inventory table, and then Amazon S3 can create a new journal or inventory table. When you re-enable the inventory table, Amazon S3 creates a new inventory table, and you're charged again for backfilling the new inventory table. + Metadata tables don't contain all the same metadata as is available through S3 Inventory or through the Amazon S3 REST API. For example, the following information isn't available in metadata tables: + S3 Lifecycle expiration eligibility or transition status + Object Lock retention period or governance mode + Object access control list (ACL) information + Replication status + When you're using Amazon Athena or Amazon Redshift to query your metadata tables, you must surround your metadata table namespace names in quotation marks (`"`) or backticks (```), otherwise the query might not work. For examples, see [Example metadata table queries](metadata-tables-example-queries.md). + When using Apache Spark on Amazon EMR or other third-party engines to query your metadata tables, we recommend that you use the Amazon S3 Tables Iceberg REST endpoint. Your query might not run successfully if you don't use this endpoint. For more information, see [Accessing tables using the Amazon S3 Tables Iceberg REST endpoint](s3-tables-integrating-open-source.md). ## S3 Metadata AWS Regions S3 Metadata is currently available in the following AWS Regions: | AWS Region name | AWS Region code | | --- | --- | | Africa (Cape Town) | `af-south-1` | | Asia Pacific (Hong Kong) | `ap-east-1` | | Asia Pacific (Jakarta) | `ap-southeast-3` | | Asia Pacific (Melbourne) | `ap-southeast-4` | | Asia Pacific (Mumbai) | `ap-south-1` | | Asia Pacific (Osaka) | `ap-northeast-3` | | Asia Pacific (Seoul) | `ap-northeast-2` | | Asia Pacific (Singapore) | `ap-southeast-1` | | Asia Pacific (Sydney) | `ap-southeast-2` | | Asia Pacific (Tokyo) | `ap-northeast-1` | | Canada (Central) | `ca-central-1` | | Canada West (Calgary) | `ca-west-1` | | Europe (Frankfurt) | `eu-central-1` | | Europe (Zurich) | `eu-central-2` | | Europe (Ireland) | `eu-west-1` | | Europe (London) | `eu-west-2` | | Europe (Milan) | `eu-south-1` | | Europe (Paris) | `eu-west-3` | | Europe (Spain) | `eu-south-2` | | Europe (Stockholm) | `eu-north-1` | | Israel (Tel Aviv) | `il-central-1` | | Middle East (Bahrain) | `me-south-1` | | Middle East (UAE) | `me-central-1` | | South America (São Paulo) | `sa-east-1` | | US East (N. Virginia) | `us-east-1` | | US East (Ohio) | `us-east-2` | | US West (N. California) | `us-west-1` | | US West (Oregon) | `us-west-2` | | China (Beijing) | `cn-north-1` | | China (Ningxia) | `cn-northwest-1` | # S3 Metadata journal tables schema The journal table records changes made to your data in near real time, helping you to identify new data uploaded to your bucket, track recently deleted objects, monitor lifecycle transitions, and more. The journal table records new objects and updates to your objects and their metadata (those updates that require either a `PUT` or a `DELETE` operation). Because this table is queryable, you can audit the changes to your bucket through simple SQL queries. You can use the journal table for security, auditing, and compliance use cases to track uploaded, deleted, and changed objects in the bucket. For example, you can query the journal table to answer questions such as: + Which objects were deleted in the past 24 hours by S3 Lifecycle? + Which IP addresses did the most recent `PUT` requests come from? + Which AWS Key Management Service (AWS KMS) keys were used for `PUT` requests in the past 7 days? + Which objects in your bucket were created by Amazon Bedrock in the last five days? Amazon S3 Metadata journal tables contain rows and columns. Each row represents a mutation event that has created, updated, or deleted an object in your general purpose bucket. Most of these events result from user actions, but some of these events result from actions taken by Amazon S3 on your behalf, such as S3 Lifecycle expirations or storage class transitions. S3 Metadata journal tables are eventually consistent with the changes that have occurred in your general purpose bucket. In some cases, by the time S3 Metadata is notified that an object is created or updated, that object might already have been overwritten or deleted in the bucket. In such cases, the objects can no longer be retrieved and some columns might show a NULL value to indicate missing metadata schema. The following is an example of a journal table for a general purpose bucket named `amzn-s3-demo-bucket:` ``` bucket key sequence_number record_type record_timestamp version_id is_delete_marker size last_modified_date e_tag storage_class is_multipart encryption_status is_bucket_key_enabled kms_key_arn checksum_algorithm object_tags user_metadata requester source_ip_address request_id amzn-s3-demo-bucket Finance/statement1.pdf 80e737d8b4d82f776affffffffffffffff006737d8b4d82f776a00000000000000000000000000000000000000000000000072 CREATE 2024-11-15 23:26:44.899 FALSE 6223 11/15/2024 23:26 e131b86632dda753aac4018f72192b83 STANDARD FALSE SSE-KMS FALSE arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890df SSECRC32 {} {count -> Asia, customs -> false, family -> true, location -> Mary, name -> football, user -> United States} 111122223333 192.0.2.1 CVK8FWYRW0M9JW65 amzn-s3-demo-bucket s3-dg.pdf 80e737d8b4e39f1dbdffffffffffffffff006737d8b4e39f1dbd00000000000000000000000000000000000000000000000072 CREATE 2024-11-15 23:26:44.942 FALSE 3554 11/15/2024 23:26 9bb49efc2d92c05558ddffbbde8636d5 STANDARD FALSE DSSE-KMS FALSE arn:aws:kms:us-east-1:936810216292:key/0dcebce6-49fd-4cae-b2e2-5512ad281afd SSESHA1 {} {} 111122223333 192.0.2.1 CVKAQDRAZEG7KXAY amzn-s3-demo-bucket Development/Projects.xls 80e737d8b4ed9ac5c6ffffffffffffffff006737d8b4ed9ac5c600000000000000000000000000000000000000000000000072 CREATE 2024-11-15 23:26:44.966 FALSE 7746 11/15/2024 23:26 729a6863e47fb9955b31bfabce984908 STANDARD FALSE SSE-S3 FALSE NULL SSECRC32 {} {count -> Asia, customs -> Canada, family -> Billiards, filter -> true, location -> Europe, name -> Asia, user -> United States} 111122223333 192.0.2.1 CVK7Z6XQTQ90BSRV ``` Journal tables have the following schema: | Column name | Required? | Data type | | | --- | --- | --- | --- | | `bucket` | Yes | String | The general purpose bucket name. For more information, see [General purpose bucket naming rules](bucketnamingrules.md). | | `key` | Yes | String | The object key name (or key) that uniquely identifies the object in the bucket. For more information, see [Naming Amazon S3 objects](object-keys.md). | | `sequence_number` | Yes | String | The sequence number, which is an ordinal that's included in the records for a given object. To order records of the same bucket and key, you can sort on `sequence_number`. For a given bucket and key, a lexicographically larger `sequence_number` value implies that the record was introduced to the bucket more recently. | | `record_type` | Yes | String | The type of this record, one of `CREATE`, `UPDATE_METADATA`, or `DELETE`. `CREATE` records indicate that a new object (or a new version of the object) was written to the bucket. `UPDATE_METADATA` records capture changes to mutable metadata for an existing object, such as the storage class or tags. `DELETE` records indicate that this object (or this version of the object) has been deleted. When versioning is enabled, `DELETE` records represent either a delete marker or a permanent delete. They are further disambiguated by consulting the optional `is_delete_marker` column. For more information, see [Deleting object versions from a versioning-enabled bucket](DeletingObjectVersions.md). A permanent delete carries `NULL`s in all columns, *except* `bucket`, `key`, `sequence_number`, `record_type`, `record_timestamp`, and `version_id` (i.e. those columns marked as Required). | | `record_timestamp` | Yes | Timestamp NTZ (no time zone) | The timestamp that's associated with this record. | | `version_id` | No | String | The object's version ID. When you enable versioning on a bucket, Amazon S3 assigns a version number to objects that are added to the bucket. For more information, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). Objects that are stored in your bucket before you set the versioning state have a version ID of null. | | `is_delete_marker` | No | Boolean | The object's delete marker status. For DELETE records that are delete markers, this value is `TRUE`. For permanent deletions, this value is omitted (`NULL`). Other record types (CREATE and UPDATE\$1METADATA) have value `FALSE`. For more information, see [Working with delete markers](DeleteMarker.md). Rows that are added for delete markers have a `record_type` value of `DELETE`, not `UPDATE_METADATA`. If the delete marker is created as the result of an S3 Lifecycle expiration, the `requester` value is `s3.amazonaws.com`. | | `size` | No | Long | The object size in bytes, not including the size of incomplete multipart uploads or object metadata. If `is_delete_marker` is `TRUE`, the size is `0`. For more information, see [System-defined object metadata](UsingMetadata.md#SysMetadata). | | `last_modified_date` | No | Timestamp NTZ (no time zone) | The object creation date or the last modified date, whichever is the latest. For multipart uploads, the object creation date is the date when the multipart upload is initiated. For more information, see [System-defined object metadata](UsingMetadata.md#SysMetadata). | | `e_tag` | No | String | The entity tag (ETag), which is a hash of the object. The ETag reflects changes only to the contents of an object, not to its metadata. The ETag can be an MD5 digest of the object data. Whether the ETag is an MD5 digest depends on how the object was created and how it's encrypted. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html) in the *Amazon S3 API Reference*. | | `storage_class` | No | String | The storage class that’s used for storing the object. One of `STANDARD`, `REDUCED_REDUNDANCY`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `DEEP_ARCHIVE`, or `GLACIER_IR`. For more information, see [Understanding and managing Amazon S3 storage classes](storage-class-intro.md). | | `is_multipart` | No | Boolean | The object's upload type. If the object was uploaded as a multipart upload, this value is `TRUE`. Otherwise, it's `FALSE`. For more information, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). | | `encryption_status` | No | String | The object's server-side encryption status, depending on what kind of encryption key is used: server-side encryption with Amazon S3 managed keys (SSE-S3), server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS), dual-layer server-side encryption with AWS KMS keys (DSSE-KMS), or server-side encryption with customer-provided keys (SSE-C). If the object is unencrypted, this value is null. Possible values are `SSE-S3`, `SSE-KMS`, `DSSE-KMS`, `SSE-C`, or null. For more information, see [Protecting data with encryption](UsingEncryption.md). | | `is_bucket_key_enabled` | No | Boolean | The object's S3 Bucket Key enablement status. If the object uses an S3 Bucket Key for SSE-KMS, this value is `TRUE`. Otherwise, it's `FALSE`. For more information, see [Configuring an S3 Bucket Key at the object level](configuring-bucket-key-object.md). | | `kms_key_arn` | No | String | The Amazon Resource Name (ARN) for the KMS key with which the object is encrypted, for rows where `encryption_status` is `SSE-KMS` or `DSSE-KMS`. If the object isn't encrypted with SSE-KMS or DSSE-KMS, the value is null. For more information, see [Using server-side encryption with AWS KMS keys (SSE-KMS)](UsingKMSEncryption.md) and [Using dual-layer server-side encryption with AWS KMS keys (DSSE-KMS)](UsingDSSEncryption.md). If a row represents an object version that no longer existed at the time that a delete or overwrite event was processed, `kms_key_arn` contains a null value, even if the `encryption_status` column value is `SSE-KMS` or `DSSE-KMS`. | | `checksum_algorithm` | No | String | The algorithm that’s used to create the checksum for the object, one of `CRC64NVME`, `CRC32`, `CRC32C`, `SHA1`, or `SHA256`. If no checksum is present, this value is null. For more information, see [Using supported checksum algorithms](checking-object-integrity-upload.md#using-additional-checksums). | | `object_tags` | No | Map | The object tags that are associated with the object. Object tags are stored as a map of key-value pairs. If an object has no object tags, an empty map (`{}`) is stored. For more information, see [Categorizing your objects using tags](object-tagging.md). If the `record_type` value is `DELETE`, the `object_tags` column contains a null value. If the `record_type` value is `CREATE` or `UPDATE_METADATA`, rows that represent object versions that no longer existed at the time that a delete or overwrite event was processed will contain a null value in the `object_tags` column. | | `user_metadata` | No | Map | The user metadata that's associated with the object. User metadata is stored as a map of key-value pairs. If an object has no user metadata, an empty map (`{}`) is stored. For more information, see [User-defined object metadata](UsingMetadata.md#UserMetadata). If the `record_type` value is `DELETE`, the `user_metadata` column contains a null value. If the `record_type` value is `CREATE` or `UPDATE_METADATA`, rows that represent object versions that no longer existed at the time that a delete or overwrite event was processed will contain a null value in the `user_metadata` column. | | `requester` | No | String | The AWS account ID of the requester or the AWS service principal that made the request. For example, if the requester is S3 Lifecycle, this value is `s3.amazonaws.com`. | | `source_ip_address` | No | String | The source IP address of the request. For records that are generated by a user request, this column contains the source IP address of the request. For actions taken by Amazon S3 or another AWS service on behalf of the user, this column contains a null value. | | `request_id` | No | String | The request ID that's associated with the request. | # S3 Metadata live inventory tables schema The live inventory table provides a simple, queryable inventory of all the objects and their versions in your bucket so that you can determine the latest state of your data. Updates to your objects are typically reflected in the inventory table within one hour. You can use this table to simplify and speed up business workflows and big data jobs by identifying objects that you want to process for various workloads. For example, you can query the inventory table to do the following: + Find all objects stored in the S3 Glacier Deep Archive storage class. + Create a distribution of object tags or find objects without tags. + Find all objects that aren't encrypted by using server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). When you enable the inventory table for your metadata table configuration, the table goes through a process known as *backfilling*, during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the inventory table within one hour. **Note** You're charged for backfilling your inventory table. If your general purpose bucket has more than one billion objects, you're also charged a monthly fee for your inventory table. For more information, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). Amazon S3 Metadata inventory tables contain rows and columns. Each row represents the current state of an object in your general purpose bucket. The inventory table provides a simple, queryable inventory of all objects in your bucket so that you can determine the current state of your data. The following is an example of an inventory table for a general purpose bucket named `amzn-s3-demo-bucket:` ``` bucket key sequence_number version_id is_delete_marker size last_modified_date e_tag storage_class is_multipart encryption_status is_bucket_key_enabled kms_key_arn checksum_algorithm object_tags user_metadata amzn-s3-demo-bucket Finance/statement1.pdf 80e737d8b4d82f776affffffffffffffff006737d8b4d82f776a00000000000000000000000000000000000000000000000072 FALSE 6223 11/15/2024 23:26 e131b86632dda753aac4018f72192b83 STANDARD FALSE SSE-KMS FALSE arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890df SSECRC32 {} {count -> Asia, customs -> false, family -> true, location -> Mary, name -> football, user -> United States} amzn-s3-demo-bucket s3-dg.pdf 80e737d8b4e39f1dbdffffffffffffffff006737d8b4e39f1dbd00000000000000000000000000000000000000000000000072 FALSE 3554 11/15/2024 23:26 9bb49efc2d92c05558ddffbbde8636d5 STANDARD FALSE DSSE-KMS FALSE arn:aws:kms:us-east-1:936810216292:key/0dcebce6-49fd-4cae-b2e2-5512ad281afd SSESHA1 {} {} amzn-s3-demo-bucket Development/Projects.xls 80e737d8b4ed9ac5c6ffffffffffffffff006737d8b4ed9ac5c600000000000000000000000000000000000000000000000072 FALSE 7746 11/15/2024 23:26 729a6863e47fb9955b31bfabce984908 STANDARD FALSE SSE-S3 FALSE NULL SSECRC32 {} {count -> Asia, customs -> Canada, family -> Billiards, filter -> true, location -> Europe, name -> Asia, user -> United States} ``` Inventory tables have the following schema: | Column name | Required? | Data type | | | --- | --- | --- | --- | | `bucket` | Yes | String | The general purpose bucket name. For more information, see [General purpose bucket naming rules](bucketnamingrules.md). | | `key` | Yes | String | The object key name (or key) that uniquely identifies the object in the bucket. For more information, see [Naming Amazon S3 objects](object-keys.md). | | `sequence_number` | Yes | String | The sequence number, which is an ordinal that's included in the records for a given object. To order records of the same bucket and key, you can sort on `sequence_number`. For a given bucket and key, a lexicographically larger `sequence_number` value implies that the record was introduced to the bucket more recently. | | `version_id` | No | String | The object's version ID. When you enable versioning on a bucket, Amazon S3 assigns a version number to objects that are added to the bucket. For more information, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). Objects that are stored in your bucket before you set the versioning state have a version ID of null. | | `is_delete_marker` | No | Boolean | The object's delete marker status. If the object is a delete marker, this value is `True`. Otherwise, it's `False`. For more information, see [Working with delete markers](DeleteMarker.md). Rows that are added for delete markers have a `record_type` value of `DELETE`, not `UPDATE_METADATA`. If the delete marker is created as the result of an S3 Lifecycle expiration, the `requester` value is `s3.amazonaws.com`. | | `size` | No | Long | The object size in bytes, not including the size of incomplete multipart uploads or object metadata. If `is_delete_marker` is `True`, the size is `0`. For more information, see [System-defined object metadata](UsingMetadata.md#SysMetadata). | | `last_modified_date` | No | Timestamp NTZ (no time zone) | The object creation date or the last modified date, whichever is the latest. For multipart uploads, the object creation date is the date when the multipart upload is initiated. For more information, see [System-defined object metadata](UsingMetadata.md#SysMetadata). | | `e_tag` | No | String | The entity tag (ETag), which is a hash of the object. The ETag reflects changes only to the contents of an object, not to its metadata. The ETag can be an MD5 digest of the object data. Whether the ETag is an MD5 digest depends on how the object was created and how it's encrypted. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html) in the *Amazon S3 API Reference*. | | `storage_class` | No | String | The storage class that’s used for storing the object. One of `STANDARD`, `REDUCED_REDUNDANCY`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `DEEP_ARCHIVE`, or `GLACIER_IR`. For more information, see [Understanding and managing Amazon S3 storage classes](storage-class-intro.md). | | `is_multipart` | No | Boolean | The object's upload type. If the object was uploaded as a multipart upload, this value is `True`. Otherwise, it's `False`. For more information, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). | | `encryption_status` | No | String | The object's server-side encryption status, depending on what kind of encryption key is used: server-side encryption with Amazon S3 managed keys (SSE-S3), server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS), dual-layer server-side encryption with AWS KMS keys (DSSE-KMS), or server-side encryption with customer-provided keys (SSE-C). If the object is unencrypted, this value is null. Possible values are `SSE-S3`, `SSE-KMS`, `DSSE-KMS`, `SSE-C`, or null. For more information, see [Protecting data with encryption](UsingEncryption.md). | | `is_bucket_key_enabled` | No | Boolean | The object's S3 Bucket Key enablement status. If the object uses an S3 Bucket Key for SSE-KMS, this value is `True`. Otherwise, it's `False`. For more information, see [Configuring an S3 Bucket Key at the object level](configuring-bucket-key-object.md). | | `kms_key_arn` | No | String | The Amazon Resource Name (ARN) for the KMS key with which the object is encrypted, for rows where `encryption_status` is `SSE-KMS` or `DSSE-KMS`. If the object isn't encrypted with SSE-KMS or DSSE-KMS, the value is null. For more information, see [Using server-side encryption with AWS KMS keys (SSE-KMS)](UsingKMSEncryption.md) and [Using dual-layer server-side encryption with AWS KMS keys (DSSE-KMS)](UsingDSSEncryption.md). If a row represents an object version that no longer existed at the time that a delete or overwrite event was processed, `kms_key_arn` contains a null value, even if the `encryption_status` column value is `SSE-KMS` or `DSSE-KMS`. | | `checksum_algorithm` | No | String | The algorithm that’s used to create the checksum for the object, one of `CRC64-NVME`, `CRC32`, `CRC32C`, `SHA1`, or `SHA256`. If no checksum is present, this value is null. For more information, see [Using supported checksum algorithms](checking-object-integrity-upload.md#using-additional-checksums). | | `object_tags` | No | Map | The object tags that are associated with the object. Object tags are stored as a map of key-value pairs. If an object has no object tags, an empty map (`{}`) is stored. For more information, see [Categorizing your objects using tags](object-tagging.md). If the `record_type` value is `DELETE`, the `object_tags` column contains a null value. If the `record_type` value is `CREATE` or `UPDATE_METADATA`, rows that represent object versions that no longer existed at the time that a delete or overwrite event was processed will contain a null value in the `object_tags` column. | | `user_metadata` | No | Map | The user metadata that's associated with the object. User metadata is stored as a map of key-value pairs. If an object has no user metadata, an empty map (`{}`) is stored. For more information, see [User-defined object metadata](UsingMetadata.md#UserMetadata). If the `record_type` value is `DELETE`, the `user_metadata` column contains a null value. If the `record_type` value is `CREATE` or `UPDATE_METADATA`, rows that represent object versions that no longer existed at the time that a delete or overwrite event was processed will contain a null value in the `user_metadata` column. | # Configuring metadata tables Amazon S3 Metadata accelerates data discovery by automatically capturing metadata for the objects in your general purpose buckets and storing it in read-only, fully managed Apache Iceberg tables that you can query. These read-only tables are called *metadata tables*. As objects are added to, updated, and removed from your general purpose buckets, S3 Metadata automatically refreshes the corresponding metadata tables to reflect the latest changes. With S3 Metadata, you can easily find, store, and query metadata for your S3 objects, so that you can quickly prepare data for use in business analytics, artificial intelligence and machine learning (AI/ML) model training, and more. To generate and store object metadata in AWS managed metadata tables, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata tables to reflect the latest changes to your data as long as the configuration is active on the bucket. Additionally, Amazon S3 continuously optimizes your metadata tables to help reduce storage costs and improve analytics query performance. To create a metadata table configuration, make sure that you have the necessary AWS Identity and Access Management (IAM) permissions to create and manage metadata tables. To monitor updates to your metadata table configuration, you can use AWS CloudTrail. For more information, see [Amazon S3 bucket-level actions that are tracked by CloudTrail logging](cloudtrail-logging-s3-info.md#cloudtrail-bucket-level-tracking). **Topics** + [ # Setting up permissions for configuring metadata tables ](metadata-tables-permissions.md) + [ # Creating metadata table configurations ](metadata-tables-create-configuration.md) + [ # Controlling access to metadata tables ](metadata-tables-access-control.md) + [ # Expiring journal table records ](metadata-tables-expire-journal-table-records.md) + [ # Enabling or disabling live inventory tables ](metadata-tables-enable-disable-inventory-tables.md) + [ # Viewing metadata table configurations ](metadata-tables-view-configuration.md) + [ # Deleting metadata table configurations ](metadata-tables-delete-configuration.md) + [ # Deleting metadata tables ](metadata-tables-delete-table.md) # Setting up permissions for configuring metadata tables To create a metadata table configuration, you must have the necessary AWS Identity and Access Management (IAM) permissions to both create and manage your metadata table configuration and to create and manage your metadata tables and the table bucket where your metadata tables are stored. To create and manage your metadata table configuration, you must have these permissions: + `s3:CreateBucketMetadataTableConfiguration` – This permission allows you to create a metadata table configuration for your general purpose bucket. To create a metadata table configuration, additional permissions, including S3 Tables permissions, are required, as explained in the following sections. For a summary of the required permissions, see [Bucket operations and permissions](using-with-s3-policy-actions.md#using-with-s3-policy-actions-related-to-buckets). + `s3:GetBucketMetadataTableConfiguration` – This permission allows you to retrieve information about your metadata table configuration. + `s3:DeleteBucketMetadataTableConfiguration` – This permission allows you to delete your metadata table configuration. + `s3:UpdateBucketMetadataJournalTableConfiguration` – This permission allows you to update your journal table configuration to expire journal table records. + `s3:UpdateBucketMetadataInventoryTableConfiguration` – This permission allows you to update your inventory table configuration to enable or disable the inventory table. To update an inventory table configuration, additional permissions, including S3 Tables permissions, are required. For a list of the required permissions, see [Bucket operations and permissions](using-with-s3-policy-actions.md#using-with-s3-policy-actions-related-to-buckets). **Note** The `s3:CreateBucketMetadataTableConfiguration`, `s3:GetBucketMetadataTableConfiguration`, and `s3:DeleteBucketMetadataTableConfiguration` permissions are used for both V1 and V2 S3 Metadata configurations. For V2, the names of the corresponding API operations are `CreateBucketMetadataConfiguration`, `GetBucketMetadataConfiguration`, and `DeleteBucketMetadataConfiguration`. To create and work with tables and table buckets, you must have certain `s3tables` permissions. At a minimum, to create a metadata table configuration, you must have the following `s3tables` permissions: + `s3tables:CreateTableBucket` – This permission allows you to create an AWS managed table bucket. All metadata table configurations in your account and in the same Region are stored in a single AWS managed table bucket named `aws-s3`. For more information, see [How metadata tables work](metadata-tables-overview.md#metadata-tables-how-they-work) and [Working with AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html). + `s3tables:CreateNamespace` – This permission allows you to create a namespace in a table bucket. Metadata tables typically use the `b_general_purpose_bucket_name` namespace. For more information about metadata table namespaces, see [How metadata tables work](metadata-tables-overview.md#metadata-tables-how-they-work). + `s3tables:CreateTable` – This permission allows you to create your metadata tables. + `s3tables:GetTable` – This permission allows you to retrieve information about your metadata tables. + `s3tables:PutTablePolicy` – This permission allows you to add or update your metadata table policies. + `s3tables:PutTableEncryption` – This permission allows you to set server-side encryption for your metadata tables. Additional permissions are required if you want to encrypt your metadata tables with server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). For more information, see [Permissions for SSE-KMS](#metadata-kms-permissions). + `kms:DescribeKey` – This permission allows you to retrieve information about a KMS key. + `s3tables:PutTableBucketPolicy` – This permission allows you to create or update a new table bucket policy. For detailed information about all table and table bucket permissions, see [Access management for S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-setting-up.html). **Important** If you also want to integrate your table bucket with AWS analytics services so that you can query your metadata table, you need additional permissions. For more information, see [Integrating Amazon S3 Tables with AWS analytics services](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html). **Permissions for SSE-KMS** To encrypt your metadata tables with server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS), you must have additional permissions. 1. The user or AWS Identity and Access Management (IAM) role needs the following permissions. You can grant these permissions by using the IAM console: [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. `s3tables:PutTableEncryption` to configure table encryption 1. `kms:DescribeKey` on the AWS KMS key used 1. On the resource policy for the KMS key, you need the following permissions. You can grant these permissions by using the AWS KMS console: [https://console.aws.amazon.com/kms](https://console.aws.amazon.com/kms). 1. Grant `kms:GenerateDataKey` permission to `metadata.s3.amazonaws.com` and `maintenance.s3tables.amazonaws.com`. 1. Grant `kms:Decrypt` permission to `metadata.s3.amazonaws.com` and `maintenance.s3tables.amazonaws.com`. 1. Grant `kms:DescribeKey` permission to the invoking AWS principal. In addition to these permissions, make sure that the customer managed KMS key used to encrypt the tables still exists, is active, is in the same Region as your general purpose bucket. **Example policy** To create and work with metadata tables and table buckets, you can use the following example policy. In this policy, the general purpose bucket that you're applying the metadata table configuration to is referred to as `amzn-s3-demo-bucket`. To use this policy, replace the `user input placeholders` with your own information. When you create your metadata table configuration, your metadata tables are stored in an AWS managed table bucket. All metadata table configurations in your account and in the same Region are stored in a single AWS managed table bucket named `aws-s3`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "PermissionsToWorkWithMetadataTables", "Effect": "Allow", "Action": [ "s3:CreateBucketMetadataTableConfiguration", "s3:GetBucketMetadataTableConfiguration", "s3:DeleteBucketMetadataTableConfiguration", "s3:UpdateBucketMetadataJournalTableConfiguration", "s3:UpdateBucketMetadataInventoryTableConfiguration", "s3tables:*", "kms:DescribeKey" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket", "arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3", "arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3/table/*" ] } ] } ``` ------ To query metadata tables, you can use the following example policy. If your metadata tables have been encrypted with SSE-KMS, you will need the `kms:Decrypt` permission as shown. To use this policy, replace the `user input placeholders` with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "PermissionsToQueryMetadataTables", "Effect": "Allow", "Action": [ "s3tables:GetTable", "s3tables:GetTableData", "s3tables:GetTableMetadataLocation", "kms:Decrypt" ], "Resource": [ "arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3", "arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3/table/*" ] } ] } ``` ------ # Creating metadata table configurations To generate and store Amazon S3 Metadata in fully managed Apache Iceberg metadata tables, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata tables to reflect the latest changes to your data as long as the configuration is active on the bucket. Additionally, Amazon S3 continuously optimizes your metadata tables to help reduce storage costs and improve analytics query performance. For each general purpose bucket, you can create a metadata table configuration that contains two complementary metadata tables: + **Journal table** – By default, your metadata table configuration contains a *journal table*, which captures events that occur for the objects in your bucket. The journal table records changes made to your data in near real time, helping you to identify new data uploaded to your bucket, track recently deleted objects, monitor lifecycle transitions, and more. The journal table records new objects and updates to your objects and their metadata (those updates that require either a `PUT` or a `DELETE` operation). The journal table captures metadata only for change events (such as uploads, updates, and deletes) that happen after you create your metadata table configuration. Because this table is queryable, you can audit the changes to your bucket through simple SQL queries. The journal table is required for each metadata table configuration. (In the initial release of S3 Metadata, the journal table was referred to as "the metadata table.") For more information about what data is stored in journal tables, see [S3 Metadata journal tables schema](metadata-tables-schema.md). To help minimize your storage costs, you can choose to enable journal table record expiration. For more information, see [Expiring journal table records](metadata-tables-expire-journal-table-records.md). + **Live inventory table** – Optionally, you can add a *live inventory table* to your metadata table configuration. The live inventory table provides a simple, queryable inventory of all the objects and their versions in your bucket so that you can determine the latest state of your data. You can use the live inventory table to simplify and speed up business workflows and big data jobs by identifying objects that you want to process for various workloads. For example, you can query the live inventory table to find all objects stored in a particular storage class, all objects with certain tags, all objects that aren't encrypted with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), and more. When you enable the live inventory table for your metadata table configuration, the table goes through a process known as *backfilling*, during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects that exist in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your live inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the live inventory table within one hour. You're charged for backfilling your live inventory table. If your general purpose bucket has more than one billion objects, you're also charged a monthly fee for your live inventory table. For more information, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). For more information about what data is stored in live inventory tables, see [S3 Metadata live inventory tables schema](metadata-tables-inventory-schema.md). Metadata tables have the following Amazon Resource Name (ARN) format, which includes the table ID of the metadata table: `arn:aws:s3tables:region-code:account-id:bucket/aws-s3/table/table-id` For example, a metadata table in the US East (N. Virginia) Region would have an ARN like the following: `arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3/table/a12bc345-67d8-912e-3456-7f89123g4h56` Journal tables have the name `journal`, and live inventory tables have the name `inventory`. When you create your metadata table configuration, your metadata tables are stored in an AWS managed table bucket. All metadata table configurations in your account and in the same Region are stored in a single AWS managed table bucket. These AWS managed table buckets are named `aws-s3` and have the following Amazon Resource Name (ARN) format: `arn:aws:s3tables:region:account_id:bucket/aws-s3` For example, if your account ID is 123456789012 and your general purpose bucket is in US East (N. Virginia) (`us-east-1`), your AWS managed table bucket is also created in US East (N. Virginia) (`us-east-1`) and has the following ARN: `arn:aws:s3tables:us-east-1:123456789012:bucket/aws-s3` By default, AWS managed table buckets are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). After you create your first metadata configuration, you can set the default encryption setting for the AWS managed table bucket to use server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). For more information, see [Encryption for AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html#aws-managed-buckets-encryption) and [Specifying server-side encryption with AWS KMS keys (SSE-KMS) in table buckets](s3-tables-kms-specify.md). Within your AWS managed table bucket, the metadata tables for your configuration are typically stored in a namespace with the following naming format: `b_general-purpose-bucket-name` For more information about metadata table namespaces, see [How metadata tables work](metadata-tables-overview.md#metadata-tables-how-they-work). When you create your metadata table configuration, you can choose to encrypt your AWS managed metadata tables with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. You can set the encryption type for your tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. To specify SSE-KMS for your metadata tables, you must have certain permissions. For more information, see [ Permissions for SSE-KMS](metadata-tables-permissions.md#metadata-kms-permissions). The encryption setting for a metadata table takes precedence over the default bucket-level encryption setting. If you don't specify encryption for a table, it will inherit the default encryption setting from the bucket. AWS managed table buckets don't count toward your S3 Tables quotas. For more information about working with AWS managed table buckets and AWS managed tables, see [Working with AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html). You can create a metadata table configuration by using the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. **Note** If you created your S3 Metadata configuration before July 15, 2025, we recommend that you delete and re-create your configuration so that you can expire journal table records and create an inventory table. For more information, see [Enabling inventory tables on metadata configurations created before July 15, 2025](#metadata-tables-migration). If you've deleted your metadata table configuration and want to re-create a configuration for the same general purpose bucket, you must first manually delete the old journal and inventory tables from your AWS managed table bucket. Otherwise, creating the new metadata table configuration fails because those tables already exist. To delete your metadata tables, see [Delete a metadata table](metadata-tables-delete-table.md#delete-metadata-table-procedure). Deleting a metadata table configuration deletes only the configuration. The AWS managed table bucket and your metadata tables still exist, even if you delete the metadata table configuration. **Prerequisites** Before you create a metadata table configuration make sure that you've met the following prerequisites: + Before you create a metadata table configuration make sure that you have the necessary AWS Identity and Access Management (IAM) permissions to create and manage metadata tables. For more information, see [Setting up permissions for configuring metadata tables](metadata-tables-permissions.md). + If you plan to query your metadata tables with Amazon Athena or another AWS query engine, make sure that you integrate your AWS managed table bucket with AWS analytics services. For more information, see [Integrating Amazon S3 Tables with AWS analytics services](s3-tables-integrating-aws.md). If you've already integrated an existing table bucket in this Region, your AWS managed table bucket is also automatically integrated. To determine the integration status for your table buckets in this Region, open the Amazon S3 console, and choose **Table buckets** in the left navigation pane. Under **Integration with AWS analytics services**, check the Region and whether the integration status says **Enabled**. ## Create a metadata table configuration ### Using the S3 console **To create a metadata table configuration** Before you create a metadata table configuration, make sure that you've reviewed and met the [prerequisites](#metadata-table-config-prereqs) and that you've reviewed [Metadata table limitations and restrictions](metadata-tables-restrictions.md). 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. Choose the general purpose bucket that you want to create a metadata table configuration for. **Note** Make sure that this general purpose bucket is an AWS Region where table buckets are available. Table buckets are available only in the US East (N. Virginia), US East (Ohio), and US West (Oregon) Regions. 1. On the bucket's details page, choose the **Metadata** tab. 1. On the **Metadata** tab, choose **Create metadata configuration**. 1. On the **Create metadata configuration** page, under **Journal table**, you can choose whether to encrypt your table with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). By default, journal tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. **Important** You can set the encryption type for your metadata tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. + To encrypt your journal table with SSE-S3 (the default), choose **Don't specify encryption type**. + To encrypt your journal table with SSE-KMS, choose **Specify encryption type**. Under **Encryption type**, choose **Server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS)**. Under **AWS KMS key**, either choose from your existing KMS keys, or enter your KMS key ARN. If you don't already have a KMS key, choose **Enter KMS key ARN**, and then choose **Create a KMS key**. Make sure that you've set up the necessary permissions for SSE-KMS. For more information, see [ Permissions for SSE-KMS](metadata-tables-permissions.md#metadata-kms-permissions). 1. (Optional) By default, the records in your journal table don't expire. To help minimize the storage costs for your journal table, choose **Enabled** for **Record expiration**. If you enable journal table record expiration, you can set the number of days to retain your journal table records. To set the **Days after which records expire** value, you can specify any whole number between `7` and `2147483647`. For example, to retain your journal table records for one year, set this value to `365`. Records will be expired within 24 to 48 hours after they become eligible for expiration. **Important** After journal table records expire, they can't be recovered. Under **Journal table records will expire after the specified number of days**, select the checkbox. 1. (Optional) If you want to add an inventory table to your metadata table configuration, under **Live inventory table**, choose **Enabled** for **Configuration status**. You can choose whether to encrypt your table with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). By default, inventory tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. **Important** You can set the encryption type for your metadata tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. + To encrypt your inventory table with SSE-S3 (the default), choose **Don't specify encryption type**. + To encrypt your inventory table with SSE-KMS, choose **Specify encryption type**. Under **Encryption type**, choose **Server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS)**. Under **AWS KMS key**, either choose from your existing KMS keys, or enter your KMS key ARN. If you don't already have a KMS key, choose **Enter KMS key ARN**, and then choose **Create a KMS key**. Make sure that you've set up the necessary permissions for SSE-KMS. For more information, see [ Permissions for SSE-KMS](metadata-tables-permissions.md#metadata-kms-permissions). 1. Choose **Create metadata table configuration**. If your metadata table configuration was successful, the names and ARNs for your metadata tables are displayed on the **Metadata** tab, along with the name of your AWS managed table bucket and namespace. If you chose to enable an inventory table for your metadata table configuration, the table goes through a process known as *backfilling*, during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects that exist in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the inventory table within one hour. To monitor updates to your metadata table configuration, you can use AWS CloudTrail. For more information, see [Amazon S3 bucket-level actions that are tracked by CloudTrail logging](cloudtrail-logging-s3-info.md#cloudtrail-bucket-level-tracking). ### Using the AWS CLI To run the following commands, you must have the AWS CLI installed and configured. If you don’t have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. Alternatively, you can run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. **To create a metadata table configuration by using the AWS CLI** Before you create a metadata table configuration, make sure that you've reviewed and met the [prerequisites](#metadata-table-config-prereqs) and that you've reviewed [Metadata table limitations and restrictions](metadata-tables-restrictions.md). To use the following example commands, replace the `user input placeholders` with your own information. 1. Create a JSON file that contains your metadata table configuration, and save it (for example, `metadata-config.json`). The following is a sample configuration. You must specify whether to enable or disable journal table record expiration. If you choose to enable record expiration, you must also specify the number of days after which your journal table records will expire. To set the `Days` value, you can specify any whole number between `7` and `2147483647`. For example, to retain your journal table records for one year, set this value to `365`. You can optionally choose to configure an inventory table. For both journal tables and inventory tables, you can optionally specify an encryption configuration. By default, metadata tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3), which you can specify by setting `SseAlgorithm` to `AES256`. To encrypt your metadata tables with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), set `SseAlgorithm` to `aws:kms`. You must also set `KmsKeyArn` to the ARN of a customer managed KMS key in the same Region where your general purpose bucket is located. ``` { "JournalTableConfiguration": { "RecordExpiration": { "Expiration": "ENABLED", "Days": 10 }, "EncryptionConfiguration": { "SseAlgorithm": "AES256" } }, "InventoryTableConfiguration": { "ConfigurationState": "ENABLED", "EncryptionConfiguration": { "SseAlgorithm": "aws:kms", "KmsKeyArn": "arn:aws:kms:us-east-2:account-id:key/key-id" } } } ``` 1. Use the following command to apply the metadata table configuration to your general purpose bucket (for example, `amzn-s3-demo-bucket`): ``` aws s3api create-bucket-metadata-configuration \ --bucket amzn-s3-demo-bucket \ --metadata-configuration file://./metadata-config.json \ --region us-east-2 ``` 1. To verify that the configuration was created, use the following command: ``` aws s3api get-bucket-metadata-configuration \ --bucket amzn-s3-demo-bucket \ --region us-east-2 ``` To monitor updates to your metadata table configuration, you can use AWS CloudTrail. For more information, see [Amazon S3 bucket-level actions that are tracked by CloudTrail logging](cloudtrail-logging-s3-info.md#cloudtrail-bucket-level-tracking). ### Using the REST API You can send REST requests to create a metadata table configuration. For more information, see [https://docs.aws.amazon.com//AmazonS3/latest/API/API_CreateBucketMetadataConfiguration.html](https://docs.aws.amazon.com//AmazonS3/latest/API/API_CreateBucketMetadataConfiguration.html) in the *Amazon S3 API Reference*. ### Using the AWS SDKs You can use the AWS SDKs to create a metadata table configuration in Amazon S3. For information, see the [list of supported SDKs](https://docs.aws.amazon.com//AmazonS3/latest/API/API_CreateBucketMetadataConfiguration.html#API_CreateBucketMetadataConfiguration_SeeAlso) in the *Amazon S3 API Reference*. ## Enabling inventory tables on metadata configurations created before July 15, 2025 If you created your S3 Metadata configuration before July 15, 2025, we recommend that you delete and re-create your configuration so that you can expire journal table records and create an inventory table. Any changes to your general purpose bucket that occur between deleting the old configuration and creating the new one aren't recorded in either of your journal tables. To migrate from an old metadata configuration to a new configuration, do the following: 1. Delete your existing metadata table configuration. For step-by-step instructions, see [Deleting metadata table configurations](metadata-tables-delete-configuration.md). 1. Create a new metadata table configuration. For step-by-step instructions, see [Creating metadata table configurations](#metadata-tables-create-configuration). If you need assistance with migrating your configuration, contact AWS Support. After you create your new metadata configuration, you will have two journal tables. If you no longer need the old journal table, you can delete it. For step-by-step instructions, see [Deleting metadata tables](metadata-tables-delete-table.md). If you've retained your old journal table and want to join it with your new one, see [Joining custom metadata with S3 metadata tables](metadata-tables-join-custom-metadata.md) for examples of how to join two tables. After migration, you can do the following: 1. To view your configuration, you can now use the `GetBucketMetadataConfiguration` API operation. To determine whether your configuration is old or new, you can look at the following attribute of your `GetBucketMetadataConfiguration` API response. An AWS managed bucket type (`"aws"`) indicates a new configuration, and a customer-managed bucket type (`"customer"`) indicates an old configuration. ``` "MetadataTableConfigurationResult": { "TableBucketType": ["aws" | "customer"] ``` For more information, see [Viewing metadata table configurations](metadata-tables-view-configuration.md). **Note** You can use the `GetBucketMetadataConfiguration` and `DeleteBucketMetadataConfiguration` API operations with old or new metadata table configurations. However, if you try to use the `GetBucketMetadataTableConfiguration` and `DeleteBucketMetadataTableConfiguration` API operations with new configurations, you will receive HTTP `405 Method Not Allowed` errors. Make sure that you update your processes to use the new API operations (`CreateBucketMetadataConfiguration`, `GetBucketMetadataConfiguration`, and `DeleteBucketMetadataConfiguration`) instead of the old API operations. 1. If you plan to query your metadata tables with Amazon Athena or another AWS query engine, make sure that you integrate your AWS managed table bucket with AWS analytics services. If you've already integrated an existing table bucket in this Region, your AWS managed table bucket is also automatically integrated. For more information, see [Integrating Amazon S3 Tables with AWS analytics services](s3-tables-integrating-aws.md). # Controlling access to metadata tables To control access to your Amazon S3 metadata tables, you can use AWS Identity and Access Management (IAM) resource-based policies that are attached to your table bucket and to your metadata tables. In other words, you can control access to your metadata tables at both the table bucket level and the table level. For more information about controlling access to your table buckets and tables, see [Access management for S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-setting-up.html). **Important** When you're creating or updating table bucket or table policies, make sure that you don't restrict the Amazon S3 service principals `metadata.s3.amazonaws.com` and `maintenance.s3tables.amazonaws.com` from writing to your table bucket or your metadata tables. If Amazon S3 is unable to write to your table bucket or your metadata tables, you must delete your metadata configuration, delete your metadata tables, and then create a new configuration. If you had an inventory table in your configuration, a new inventory table has to be created, and you will be charged again for backfilling the new inventory table. You can also control access to the rows and columns in your metadata tables through AWS Lake Formation. For more information, see [Managing Lake Formation permissions](https://docs.aws.amazon.com/lake-formation/latest/dg/managing-permissions.html) and [Data filtering and cell-level security in Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/data-filtering.html) in the *AWS Lake Formation Developer Guide*. # Expiring journal table records By default, the records in your journal table don't expire. To help minimize the storage costs for your journal table, you can enable journal table record expiration. **Note** If you created your S3 Metadata configuration before July 15, 2025, you can't enable journal table record expiration on that configuration. We recommend that you delete and re-create your configuration so that you can expire journal table records and create an inventory table. For more information, see [Enabling inventory tables on metadata configurations created before July 15, 2025](metadata-tables-create-configuration.md#metadata-tables-migration). If you enable journal table record expiration, you can set the number of days to retain your journal table records. To set this value, specify any whole number between `7` and `2147483647`. For example, to retain your journal table records for one year, set this value to `365`. **Important** After journal table records expire, they can't be recovered. Records are expired within 24 to 48 hours after they become eligible for expiration. Journal records are removed from the latest snapshot. The data and storage for the deleted records is removed through table maintenance operations. If you've enabled journal table record expiration, you can disable it at any time to stop expiring your journal table records. You can expire journal table records by using the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. ## How to expire journal table records ### Using the S3 console **To expire journal table records** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. Choose the general purpose bucket that contains the metadata table configuration with the journal table that you want to expire records from. 1. On the bucket's details page, choose the **Metadata** tab. 1. On the **Metadata** tab, choose **Edit**, then choose **Edit journal table record expiration**. 1. On the **Edit journal table record expiration** page, choose **Enabled** under **Record expiration**. 1. Set the number of days to retain your journal table records. To set the **Days after which records expire** value, specify any whole number between `7` and `2147483647`. For example, to retain your journal table records for one year, set this value to `365`. **Important** After journal table records expire, they can't be recovered. 1. Under **Journal table records will expire after the specified number of days**, select the checkbox. 1. Choose **Save changes**. If you want to disable journal table record expiration, repeat the preceding steps, but choose **Disabled** instead of **Enabled** for step 6. ### Using the AWS CLI To run the following commands, you must have the AWS CLI installed and configured. If you don't have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. You can also run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. **To expire journal table records by using the AWS CLI** To use the following example commands, replace the `user input placeholders` with your own information. 1. Create a JSON file that contains your journal table configuration, and save it (for example, `journal-config.json`). The following is a sample configuration. To set the `Days` value, specify any whole number between `7` and `2147483647`. For example, to retain your journal table records for one year, set this value to `365`. ``` { "RecordExpiration": { "Expiration": "ENABLED", "Days": 10 } } ``` To disable journal table record expiration, create the following sample configuration instead. If `Expiration` is set to `DISABLED`, you must not specify a `Days` value in the configuration. ``` { "RecordExpiration": { "Expiration": "DISABLED" } } ``` 1. Use the following command to expire records from the journal table in your general purpose bucket (for example, `amzn-s3-demo-bucket`): ``` aws s3api update-bucket-metadata-journal-table-configuration \ --bucket amzn-s3-demo-bucket \ --journal-table-configuration file://./journal-config.json \ --region us-east-2 ``` ### Using the REST API You can send REST requests to expire journal table records. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_UpdateBucketMetadataJournalTableConfiguration.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UpdateBucketMetadataJournalTableConfiguration.html). ### Using the AWS SDKs You can use the AWS SDKs to expire journal table records in Amazon S3. For information, see the [list of supported SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UpdateBucketMetadataJournalTableConfiguration.html#API_UpdateBucketMetadataJournalTableConfiguration_SeeAlso). # Enabling or disabling live inventory tables By default, your metadata table configuration contains a *journal table*, which records the events that occur for the objects in your bucket. The journal table is required for each metadata table configuration. Optionally, you can add a *live inventory table* to your metadata table configuration. The live inventory table provides a simple, queryable inventory of all the objects and their versions in your bucket so that you can determine the latest state of your data. **Note** If you created your S3 Metadata configuration before July 15, 2025, you can't enable an inventory table on that configuration. We recommend that you delete and re-create your configuration so that you can create an inventory table and expire journal table records. For more information, see [Enabling inventory tables on metadata configurations created before July 15, 2025](metadata-tables-create-configuration.md#metadata-tables-migration). The inventory table contains the latest metadata for all objects in your bucket. You can use this table to simplify and speed up business workflows and big data jobs by identifying objects that you want to process for various workloads. For example, you can query the inventory table to do the following: + Find all objects stored in the S3 Glacier Deep Archive storage class. + Create a distribution of object tags or find objects without tags. + Find all objects that aren't encrypted by using server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). + Compare your inventory table at two different points in time to understand the growth in objects with specific tags. If you chose to enable an inventory table for your metadata table configuration, the table goes through a process known as *backfilling*, during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects that exist in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the inventory table within one hour. **Note** You're charged for backfilling your inventory table. If your general purpose bucket has more than one billion objects, you're also charged a monthly fee for your inventory table. For more information, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). You can't pause updates to your inventory table and then resume them. However, you can disable the inventory table configuration. Disabling the inventory table doesn't delete it. The inventory table is retained for your records until you decide to delete it. If you've disabled your inventory table and later want to re-enable it, you must first delete the old inventory table from your AWS managed table bucket. When you re-enable the inventory table configuration, Amazon S3 creates a new inventory table, and you're charged again for backfilling the new inventory table. You can enable or disable inventory tables by using the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. **Prerequisites** If you've disabled your inventory table and now want to re-enable it, you must first manually delete the old inventory table from your AWS managed table bucket. Otherwise, re-enabling the inventory table fails because an inventory table already exists in the table bucket. To delete your inventory table, see [Delete a metadata table](metadata-tables-delete-table.md#delete-metadata-table-procedure). When you re-enable the inventory table configuration, Amazon S3 creates a new inventory table, and you're charged again for backfilling the new inventory table. ## Enable or disable inventory tables ### Using the S3 console **To enable or disable inventory tables** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. Choose the general purpose bucket with the metadata table configuration that you want to enable or disable an inventory table for. 1. On the bucket's details page, choose the **Metadata** tab. 1. On the **Metadata** tab, choose **Edit**, then choose **Edit inventory table configuration**. 1. On the **Edit inventory table configuration** page, choose **Enabled** or **Disabled** under **Inventory table**. **Note** Before you choose **Enabled**, make sure that you've reviewed and met the [prerequisites](#inventory-table-config-prereqs). + If you chose **Enabled**, you can choose whether to encrypt your table with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). By default, inventory tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. **Important** You can set the encryption type for your metadata tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. + To encrypt your inventory table with SSE-S3 (the default), choose **Don't specify encryption type**. + To encrypt your inventory table with SSE-KMS, choose **Specify encryption type**. Under **Encryption type**, choose **Server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS)**. Under **AWS KMS key**, either choose from your existing KMS keys, or enter your KMS key ARN. If you don't already have a KMS key, choose **Enter KMS key ARN**, and then choose **Create a KMS key**. + If you chose **Disabled**, under **After the inventory table is disabled, the table will no longer be updated, and updates can't be resumed**, select the checkbox. 1. Choose **Save changes**. ### Using the AWS CLI To run the following commands, you must have the AWS CLI installed and configured. If you don’t have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. Alternatively, you can run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. **To enable or disable inventory tables by using the AWS CLI** To use the following example commands, replace the `user input placeholders` with your own information. **Note** Before enabling an inventory configuration, make sure that you've reviewed and met the [prerequisites](#inventory-table-config-prereqs). 1. Create a JSON file that contains your inventory table configuration, and save it (for example, `inventory-config.json`). The following is a sample configuration to enable a new inventory table. If you're enabling an inventory table, you can optionally specify an encryption configuration. By default, metadata tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3), which you can specify by setting `SseAlgorithm` to `AES256`. To encrypt your inventory table with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), set `SseAlgorithm` to `aws:kms`. You must also set `KmsKeyArn` to the ARN of a customer managed KMS key in the same Region where your general purpose bucket is located. ``` { "ConfigurationState": "ENABLED", "EncryptionConfiguration": { "SseAlgorithm": "aws:kms", "KmsKeyArn": "arn:aws:kms:us-east-2:account-id:key/key-id" } } ``` If you want to disable an existing inventory table, use the following configuration: ``` { "ConfigurationState": "DISABLED" } } ``` 1. Use the following command to update the inventory table configuration for your general purpose bucket (for example, `amzn-s3-demo-bucket`): ``` aws s3api update-bucket-metadata-inventory-table-configuration \ --bucket amzn-s3-demo-source-bucket \ --inventory-table-configuration file://./inventory-config.json \ --region us-east-2 ``` ### Using the REST API You can send REST requests to enable or disable inventory tables. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_UpdateBucketMetadataInventoryTableConfiguration.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UpdateBucketMetadataInventoryTableConfiguration.html). ### Using the AWS SDKs You can use the AWS SDKs to enable or disable inventory tables in Amazon S3. For information, see the [list of supported SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UpdateBucketMetadataInventoryTableConfiguration.html#API_UpdateBucketMetadataInventoryTableConfiguration_SeeAlso). # Viewing metadata table configurations If you've created a metadata table configuration for a general purpose bucket, you can view information about the configuration, such as whether an inventory table has been enabled, or whether journal table record expiration has been enabled. You can also view the status of your journal and inventory tables. You can view your metadata table configuration for a general purpose bucket by using the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. ## View a metadata table configuration ### Using the S3 console **To view a metadata table configuration** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. Choose the general purpose bucket that contains the metadata table configuration that you want to view. 1. On the bucket's details page, choose the **Metadata** tab. 1. On the **Metadata** tab, scroll down to the **Metadata configuration** section. In the **Journal table** and **Inventory table** sections, you can view various information for these configurations, such as their Amazon Resource Names (ARNs), the status of your tables, and whether you've enabled journal table record expiration or an inventory table. ### Using the AWS CLI To run the following commands, you must have the AWS CLI installed and configured. If you don’t have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. Alternatively, you can run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. **To view a metadata table configuration by using the AWS CLI** To use the following example command, replace the `user input placeholders` with your own information. 1. Use the following command to view the metadata table configuration for your general purpose bucket (for example, `amzn-s3-demo-bucket`): ``` aws s3api get-bucket-metadata-configuration \ --bucket amzn-s3-demo-bucket \ --region us-east-2 ``` 1. View the output of this command to see the status of your metadata table configuration. For example: ``` { "GetBucketMetadataConfigurationResult": { "MetadataConfigurationResult": { "DestinationResult": { "TableBucketType": "aws", "TableBucketArn": "arn:aws:s3tables:us-east-2:111122223333:bucket/aws-managed-s3-111122223333-us-east-2", "TableNamespace": "b_general-purpose-bucket-name" }, "JournalTableConfigurationResult": { "TableStatus": "ACTIVE", "TableName": "journal", "TableArn": "arn:aws:s3tables:us-east-2:111122223333:bucket/aws-managed-s3-111122223333-us-east-2/table/0f01234c-fe7a-492f-a4c7-adec3864ea85", "EncryptionConfiguration": { "SseAlgorithm": "AES256" }, "RecordExpiration": { "Expiration": "ENABLED", "Days": 10 } }, "InventoryTableConfigurationResult": { "ConfigurationState": "ENABLED", "TableStatus": "BACKFILL_COMPLETE", "TableName": "inventory", "TableArn": "arn:aws:s3tables:us-east-2:111122223333:bucket/aws-managed-s3-111122223333-us-east-2/table/e123456-b876-4e5e-af29-bb055922ee4d", "EncryptionConfiguration": { "SseAlgorithm": "AES256" } } } } } ``` ### Using the REST API You can send REST requests to view a metadata table configuration. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketMetadataTableConfiguration.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketMetadataTableConfiguration.html). **Note** You can use the V2 `GetBucketMetadataConfiguration` API operation with V1 or V2 metadata table configurations. However, if you try to use the V1 `GetBucketMetadataTableConfiguration` API operation with V2 configurations, you will receive an HTTP `405 Method Not Allowed` error. ### Using the AWS SDKs You can use the AWS SDKs to view a metadata table configuration in Amazon S3. For information, see the [list of supported SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketMetadataTableConfiguration.html#API_GetBucketMetadataTableConfiguration_SeeAlso). # Deleting metadata table configurations If you want to stop updating the metadata table configuration for an Amazon S3 general purpose bucket, you can delete the metadata table configuration that's attached to your bucket. Deleting a metadata table configuration deletes only the configuration. The AWS managed table bucket and your metadata tables still exist, even if you delete the metadata table configuration. However, the metadata tables will no longer be updated. **Note** If you delete your metadata table configuration and want to re-create a configuration for the same general purpose bucket, you must first manually delete the old journal and inventory tables from your AWS managed table bucket. Otherwise, creating the new metadata table configuration fails because those tables already exist. To delete your metadata tables, see [Deleting metadata tables](metadata-tables-delete-table.md). To delete your metadata tables, see [Delete a metadata table](metadata-tables-delete-table.md#delete-metadata-table-procedure). To delete your table bucket, see [Deleting table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets-delete.html) and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_DeleteTableBucket.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_DeleteTableBucket.html) in the *Amazon S3 API Reference*. You can delete a metadata table configuration by using the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. ## Delete a metadata table configuration ### Using the S3 console **To delete a metadata table configuration** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. Choose the general purpose bucket that you want to remove a metadata table configuration from. 1. On the bucket's details page, choose the **Metadata** tab. 1. On the **Metadata** tab, choose **Delete**. 1. In the **Delete metadata configuration** dialog box, enter **confirm** to confirm that you want to delete the configuration. Then choose **Delete**. ### Using the AWS CLI To run the following commands, you must have the AWS CLI installed and configured. If you don’t have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. Alternatively, you can run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. **To delete a metadata table configuration by using the AWS CLI** To use the following example commands, replace the `user input placeholders` with your own information. 1. Use the following command to delete the metadata table configuration from your general purpose bucket (for example, `amzn-s3-demo-bucket`): ``` aws s3api delete-bucket-metadata-configuration \ --bucket amzn-s3-demo-bucket \ --region us-east-2 ``` 1. To verify that the configuration was deleted, use the following command: ``` aws s3api get-bucket-metadata-configuration \ --bucket amzn-s3-demo-bucket \ --region us-east-2 ``` ### Using the REST API You can send REST requests to delete a metadata table configuration. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketMetadataConfiguration.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketMetadataConfiguration.html). **Note** You can use the V2 `DeleteBucketMetadataConfiguration` API operation with V1 or V2 metadata table configurations. However, if you try to use the V1 `DeleteBucketMetadataTableConfiguration` API operation with V2 configurations, you will receive an HTTP `405 Method Not Allowed` error. ### Using the AWS SDKs You can use the AWS SDKs to delete a metadata table configuration in Amazon S3. For information, see the [list of supported SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketMetadataConfiguration.html#API_DeleteBucketMetadataConfiguration_SeeAlso). # Deleting metadata tables If you want to delete the metadata tables that you created for an Amazon S3 general purpose bucket, you can delete the metadata tables from your AWS managed table bucket. **Important** Deleting a table is permanent and can't be undone. Before deleting a table, make sure that you have backed up any important data. Before you delete a metadata table, we recommend that you first delete the associated metadata table configuration on your general purpose bucket. For more information, see [Deleting metadata table configurations](metadata-tables-delete-configuration.md). To delete your AWS managed table bucket, see [Deleting table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets-delete.html) and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_DeleteTableBucket.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_DeleteTableBucket.html) in the *Amazon S3 API Reference*. Before you delete your AWS managed table bucket, we recommend that you first delete all metadata table configurations that are associated with this bucket. You must also first delete all metadata tables in the bucket. You can delete a metadata table by using the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. ## Delete a metadata table ### Using the AWS CLI To run the following commands, you must have the AWS CLI installed and configured. If you don’t have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. Alternatively, you can run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. **To delete a metadata table configuration by using the AWS CLI** To use the following example commands, replace the `user input placeholders` with your own information. 1. Use the following command to delete the metadata table from your AWS managed table bucket: ``` aws s3tables delete-table \ --table-bucket-arn arn:aws:s3tables:us-east-2:111122223333:bucket/aws-s3 \ --namespace b_general-purpose-bucket-name \ --name journal \ --region us-east-2 ``` 1. To verify that the table was deleted, use the following command: ``` aws s3tables get-table \ --table-bucket-arn arn:aws:s3tables:us-east-2:111122223333:bucket/aws-s3 \ --namespace b_general-purpose-bucket-name \ --name journal \ --region us-east-2 ``` ### Using the REST API You can send REST requests to delete a metadata table configuration. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_DeleteTable.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_DeleteTable.html) in the *Amazon S3 API Reference*. ### Using the AWS SDKs You can use the AWS SDKs to delete a metadata table configuration in Amazon S3. For information, see the [list of supported SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/API_s3TableBuckets_DeleteTable.html#API_s3TableBuckets_DeleteTable_SeeAlso) in the *Amazon S3 API Reference*. # Querying metadata tables Your Amazon S3 Metadata tables are stored in an AWS managed S3 table bucket, which provides storage that's optimized for tabular data. To query your metadata, you can integrate your table bucket with Amazon SageMaker Lakehouse. This integration, which uses the AWS Glue Data Catalog and AWS Lake Formation, allows AWS analytics services to automatically discover and access your table data. After your table bucket is integrated with the AWS Glue Data Catalog, you can directly query your metadata tables with AWS analytics services such as Amazon Athena, Amazon EMR, and Amazon Redshift. You can also create interactive dashboards with your query data by using Amazon Quick. For more information about integrating your AWS managed S3 table bucket with Amazon SageMaker Lakehouse, see [Integrating Amazon S3 Tables with AWS analytics services](s3-tables-integrating-aws.md). You can also query your metadata tables with Apache Spark, Apache Trino, and any other application that supports the Apache Iceberg format by using the AWS Glue Iceberg REST endpoint, Amazon S3 Tables Iceberg REST endpoint, or the Amazon S3 Tables Catalog for Apache Iceberg client catalog. For more information about accessing your metadata tables, see [Accessing table data](s3-tables-access.md). You can analyze your metadata tables with any query engine that supports the Apache Iceberg format. For example, you can query your metadata tables to do the following: + Discover storage usage patterns and trends + Audit AWS Key Management Service (AWS KMS) encryption key usage across your objects + Search for objects by user-defined metadata and object tags + Understand object metadata changes over time + Learn when objects are updated or deleted, including the AWS account ID or IP address that made the request You can also join S3 managed metadata tables and custom metadata tables, allowing you to query across multiple datasets. ## Query pricing considerations Additional pricing applies for running queries on your metadata tables. For more information, see pricing information for the query engine that you're using. For information on making your queries more cost effective, see [Optimizing metadata table query performance](metadata-tables-optimizing-query-performance.md). **Topics** + [ ## Query pricing considerations ](#metadata-tables-querying-pricing) + [ # Permissions for querying metadata tables ](metadata-tables-bucket-query-permissions.md) + [ # Querying metadata tables with AWS analytics services ](metadata-tables-bucket-integration.md) + [ # Querying metadata tables with open-source query engines ](metadata-tables-bucket-integration-open-source.md) + [ # Optimizing metadata table query performance ](metadata-tables-optimizing-query-performance.md) + [ # Example metadata table queries ](metadata-tables-example-queries.md) # Permissions for querying metadata tables Before you can query your S3 Metadata journal and live inventory tables, you must have certain S3 Tables permissions. If your metadata tables have been encrypted with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), you must also have the `kms:Decrypt` permission to decrypt the table data. When you create your metadata table configuration, your metadata tables are stored in an AWS managed table bucket. All metadata table configurations in your account and in the same Region are stored in a single AWS managed table bucket named `aws-s3`. To query metadata tables, you can use the following example policy. To use this policy, replace the `user input placeholders` with your own information. ``` { "Version":"2012-10-17", "Statement":[ { "Sid":"PermissionsToQueryMetadataTables", "Effect":"Allow", "Action":[ "s3tables:GetTable", "s3tables:GetTableData", "s3tables:GetTableMetadataLocation", "kms:Decrypt" ], "Resource":[ "arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3", "arn:aws:s3tables:us-east-1:111122223333:bucket/aws-s3/table/*", "arn:aws:kms:us-east-1:111122223333:key/01234567-89ab-cdef-0123-456789abcdef" ] } ] } ``` # Querying metadata tables with AWS analytics services You can query your S3 managed metadata tables with AWS analytics services such as Amazon Athena, Amazon Redshift, and Amazon EMR. Before you can run queries, you must first [integrate the AWS managed S3 table buckets](s3-tables-integrating-aws.md) in your AWS account and Region with AWS analytics services. ## Querying metadata tables with Amazon Athena After you [integrate your AWS managed S3 table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html) with AWS analytics services, you can start querying your metadata tables in Athena. In your queries, do the following: + Specify your catalog as `s3tablescatalog/aws-s3` and your database as `b_general_purpose_bucket_name` (which is typically the namespace for your metadata tables). + Make sure to surround your metadata table namespace names in quotation marks (`"`) or backticks (```), otherwise the query might not work. For more information, see [Querying Amazon S3 tables with Athena](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-athena.html). You can also run queries in Athena from the Amazon S3 console. ### Using the S3 console and Amazon Athena The following procedure uses the Amazon S3 console to access the Athena query editor so that you can query a table with Amazon Athena. **To query a metadata table** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. On the **General purpose buckets** tab, choose the bucket that contains the metadata configuration for the metadata table that you want to query. 1. On the bucket details page, choose the **Metadata** tab. 1. Choose **Query table with Athena**, and then choose one of the sample queries for journal or inventory tables. 1. The Amazon Athena console opens and the Athena query editor appears with a sample query loaded for you. Modify this query as needed for your use case. In the query editor, the **Catalog** field should be populated with **s3tablescatalog/aws-s3**. The **Database** field should be populated with the namespace where your table is stored (for example, **b\$1*general-purpose-bucket-name***). **Note** If you don't see these values in the **Catalog** and **Database** fields, make sure that you've integrated your AWS managed table bucket with AWS analytics services in this Region. For more information, see [Integrating Amazon S3 Tables with AWS analytics services](s3-tables-integrating-aws.md). 1. To run the query, choose **Run**. **Note** If you receive the error "Insufficient permissions to execute the query. Principal does not have any privilege on specified resource" when you try to run a query in Athena, you must be granted the necessary Lake Formation permissions on the table. For more information, see [Granting Lake Formation permission on a table or database](grant-permissions-tables.md#grant-lf-table). Also make sure that you have the appropriate AWS Identity and Access Management (IAM) permissions to query metadata tables. For more information, see [Permissions for querying metadata tables](metadata-tables-bucket-query-permissions.md). If you receive the error "Iceberg cannot access the requested resource" when you try to run the query, go to the AWS Lake Formation console and make sure that you've granted yourself permissions on the table bucket catalog and database (namespace) that you created. Don't specify a table when granting these permissions. For more information, see [Granting Lake Formation permission on a table or database](grant-permissions-tables.md#grant-lf-table). ## Querying metadata tables with Amazon Redshift After you [integrate your AWS managed S3 table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html) with AWS analytics services, do the following: + [Create a resource link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html#database-link-tables) to your metadata table namespace (typically `b_general_purpose_bucket_name`). + Make sure to surround your metadata table namespace names in quotation marks (`"`) or backticks (```), otherwise the query might not work. After that's done, you can start querying your metadata tables in the Amazon Redshift console. For more information, see [Accessing Amazon S3 tables with Amazon Redshift](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-redshift.html). ## Querying metadata tables with Amazon EMR To query your metadata tables by using Amazon EMR, you create an Amazon EMR cluster configured for Apache Iceberg and connect to your metadata tables using Apache Spark. You can set this up by integrating your AWS managed S3 table buckets with AWS analytics services or using the open-source Amazon S3 Tables Catalog for Iceberg client catalog. **Note** When using Apache Spark on Amazon EMR or other third-party engines to query your metadata tables, we recommend that you use the Amazon S3 Tables Iceberg REST endpoint. Your query might not run successfully if you don't use this endpoint. For more information, see [Accessing tables using the Amazon S3 Tables Iceberg REST endpoint](s3-tables-integrating-open-source.md). For more information, see [Accessing Amazon S3 tables with Amazon EMR](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-emr.html). # Querying metadata tables with open-source query engines You can query your S3 managed metadata tables by using open-source query engines, such as Apache Spark. When using Apache Spark on Amazon EMR or other third-party engines to query your metadata tables, we recommend that you use the Amazon S3 Tables Iceberg REST endpoint. Your query might not run successfully if you don't use this endpoint. For more information, see [Accessing tables using the Amazon S3 Tables Iceberg REST endpoint](s3-tables-integrating-open-source.md). # Optimizing metadata table query performance Because S3 Metadata is based on the Apache Iceberg table format, you can optimize the performance and [cost](#metadata-tables-optimizing-query-performance) of your journal table queries by using specific time ranges. For example, the following SQL query provides the sensitivity level of new objects in an S3 general purpose bucket: ``` SELECT key, object_tags['SensitivityLevel'] FROM "b_general-purpose-bucket-name"."journal" WHERE record_type = 'CREATE' GROUP BY object_tags['SensitivityLevel'] ``` This query scans the entire journal table, which might take a long time to run. To improve performance, you can include the `record_timestamp` column to focus on a specific time range. We also recommend using the fully qualified table name, which you can find in the Amazon S3 console on the metadata configuration details page on the general purpose bucket's **Metadata** tab. Here's an updated version of the previous query that looks at new objects from the past month: ``` SELECT key, object_tags['SensitivityLevel'] FROM b_general-purpose-bucket-name"."aws-s3.b_general-purpose-bucket-name.journal" WHERE record_type = 'CREATE' AND record_timestamp > (CURRENT_TIMESTAMP – interval '1' month) GROUP BY object_tags['SensitivityLevel'] ``` To improve the performance of queries on inventory tables, make sure that you query only on the minimum columns that you need. # Example metadata table queries The following examples show how you can get different types information from your S3 Metadata tables by using standard SQL queries. Remember when using these examples: + The examples are written to work with Amazon Athena. You might have to modify the examples to work with a different query engine. + Make sure that you understand how to [optimize your queries](metadata-tables-optimizing-query-performance.md). + Replace `b_general-purpose-bucket-name` with the name of your namespace. + For a full list of supported columns, see the [S3 Metadata journal tables schema](metadata-tables-schema.md) and [S3 Metadata live inventory tables schema](metadata-tables-inventory-schema.md). **Contents** + [ ## Journal table example queries ](#metadata-tables-example-queries-journal-tables) + [ ### Finding objects by file extension ](#metadata-tables-example-query-object-pattern) + [ ### Listing object deletions ](#metadata-tables-example-query-delete-events) + [ ### Listing AWS KMS encryption keys used by your objects ](#metadata-tables-example-query-objects-using-kms-key) + [ ### Listing objects that don't use KMS keys ](#metadata-tables-example-query-objects-not-using-kms-key) + [ ### Listing AWS KMS encryption keys used for `PUT` operations in the last 7 days ](#metadata-tables-example-query-objects-using-kms-key-puts) + [ ### Listing objects deleted in the last 24 hours by S3 Lifecycle ](#metadata-tables-example-query-objects-deleted-lifecycle) + [ ### Viewing metadata provided by Amazon Bedrock ](#metadata-tables-example-query-bedrock) + [ ### Understanding the current state of your objects ](#metadata-tables-example-query-current-state) + [ ## Inventory table example queries ](#metadata-tables-example-queries-inventory-tables) + [ ### Discovering datasets that use specific tags ](#metadata-tables-example-query-datasets-specific-tags) + [ ### Listing objects not encrypted with SSE-KMS ](#metadata-tables-example-query-objects-not-kms-encrypted) + [ ### Listing objects that aren't encrypted ](#metadata-tables-example-query-objects-not-encrypted) + [ ### Listing objects generated by Amazon Bedrock ](#metadata-tables-example-query-objects-generated-bedrock) + [ ### Reconciling the inventory table with the journal table ](#metadata-tables-example-query-generate-latest-inventory) + [ ### Finding the current versions of your objects ](#metadata-tables-example-query-latest-version) + [ # Joining custom metadata with S3 metadata tables ](metadata-tables-join-custom-metadata.md) + [ # Visualizing metadata table data with Amazon Quick ](metadata-tables-quicksight-dashboards.md) ## Journal table example queries You can use the following example queries to query your journal tables. ### Finding objects by file extension The following query returns objects with a specific file extension (`.jpg` in this case): ``` SELECT key FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" WHERE key LIKE '%.jpg' AND record_type = 'CREATE' ``` ### Listing object deletions The following query returns object deletion events, including the AWS account ID or AWS service principal that made the request: ``` SELECT DISTINCT bucket, key, sequence_number, record_type, record_timestamp, requester, source_ip_address, version_id FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" WHERE record_type = 'DELETE'; ``` ### Listing AWS KMS encryption keys used by your objects The following query returns the ARNs of the AWS Key Management Service (AWS KMS) keys encrypting your objects: ``` SELECT DISTINCT kms_key_arn FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal"; ``` ### Listing objects that don't use KMS keys The following query returns objects that aren't encrypted with AWS KMS keys: ``` SELECT DISTINCT kms_key_arn FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" WHERE encryption_status NOT IN ('SSE-KMS', 'DSSE-KMS') AND record_type = 'CREATE'; ``` ### Listing AWS KMS encryption keys used for `PUT` operations in the last 7 days The following query returns the ARNs of the AWS Key Management Service (AWS KMS) keys encrypting your objects: ``` SELECT DISTINCT kms_key_arn FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" WHERE record_timestamp > (current_date - interval '7' day) AND kms_key_arn is NOT NULL; ``` ### Listing objects deleted in the last 24 hours by S3 Lifecycle The following query returns lists the objects expired in the last day by S3 Lifecycle: ``` SELECT bucket, key, version_id, last_modified_date, record_timestamp, requester FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" WHERE requester = 's3.amazonaws.com' AND record_type = 'DELETE' AND record_timestamp > (current_date - interval '1' day) ``` ### Viewing metadata provided by Amazon Bedrock Some AWS services (such as [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/APIReference/welcome.html)), upload objects to Amazon S3. You can query the object metadata provided by these services. For example, the following query includes the `user_metadata` column to determine if there are objects uploaded by Amazon Bedrock to a general purpose bucket: ``` SELECT DISTINCT bucket, key, sequence_number, record_type, record_timestamp, user_metadata FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" WHERE record_type = 'CREATE' AND user_metadata['content-source'] = 'AmazonBedrock'; ``` If Amazon Bedrock uploaded an object to your bucket, the `user_metadata` column will display the following metadata associated with the object in the query result: ``` user_metadata {content-additional-params -> requestid="CVK8FWYRW0M9JW65", signedContentSHA384="38b060a751ac96384cd9327eb1b1e36a21fdb71114be07434c0cc7bf63f6e1da274edebfe76f65fbd51ad2f14898b95b", content-model-id -> bedrock-model-arn, content-source -> AmazonBedrock} ``` ### Understanding the current state of your objects The following query can help you determine the current state of your objects. The query identifies the most recent version of each object, filters out deleted objects, and marks the latest version of each object based on sequence numbers. Results are ordered by the `bucket`, `key`, and `sequence_number` columns. ``` WITH records_of_interest as ( -- Start with a query that can narrow down the records of interest. SELECT * from "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" ), version_stacks as ( SELECT *, -- Introduce a column called 'next_sequence_number', which is the next larger -- sequence_number for the same key version_id in sorted order. LEAD(sequence_number, 1) over (partition by (bucket, key, coalesce(version_id, '')) order by sequence_number ASC) as next_sequence_number from records_of_interest ), -- Pick the 'tip' of each version stack triple: (bucket, key, version_id). -- The tip of the version stack is the row of that triple with the largest sequencer. -- Selecting only the tip filters out any row duplicates. -- This isn't typical, but some events can be delivered more than once to the table -- and include rows that might no longer exist in the bucket (since the -- table contains rows for both extant and extinct objects). -- In the next subquery, eliminate the rows that contain deleted objects. current_versions as ( SELECT * from version_stacks where next_sequence_number is NULL ), -- Eliminate the rows that are extinct from the bucket by filtering with -- record_type. An object version has been deleted from the bucket if its tip is -- record_type==DELETE. existing_current_versions as ( SELECT * from current_versions where not (record_type = 'DELETE' and is_delete_marker = FALSE) ), -- Optionally, to determine which of several object versions is the 'latest', -- you can compare their sequence numbers. A version_id is the latest if its -- tip's sequencer is the largest among all other tips in the same key. with_is_latest as ( SELECT *, -- Determine if the sequence_number of this row is the same as the largest sequencer for the key that still exists. sequence_number = (MAX(sequence_number) over (partition by (bucket, key))) as is_latest_version FROM existing_current_versions ) SELECT * from with_is_latest ORDER BY bucket, key, sequence_number; ``` ## Inventory table example queries You can use the following example queries to query your inventory tables. ### Discovering datasets that use specific tags The following query returns the dataset that uses the specified tags: ``` SELECT * FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."inventory" WHERE object_tags['key1'] = 'value1' AND object_tags['key2'] = 'value2'; ``` ### Listing objects not encrypted with SSE-KMS The following query returns objects that aren't encrypted with SSE-KMS: ``` SELECT key, encryption_status FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."inventory" WHERE encryption_status != 'SSE-KMS'; ``` ### Listing objects that aren't encrypted The following query returns objects that aren't encrypted: ``` SELECT bucket, key, version_id FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."inventory" WHERE encryption_status IS NULL; ``` ### Listing objects generated by Amazon Bedrock The following query lists objects that were generated by Amazon Bedrock: ``` SELECT DISTINCT bucket, key, sequence_number, user_metadata FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."inventory" WHERE user_metadata['content-source'] = 'AmazonBedrock'; ``` ### Reconciling the inventory table with the journal table The following query generates an inventory-table-like list that's up to date with the current contents of the bucket. More precisely, the resulting list combines the latest snapshot of the inventory table with the latest events in the journal table. For this query to produce the most accurate results, both the journal and inventory tables must be in Active status. We recommend using this query for general purpose buckets containing fewer than a billion (10^9) objects. This example query applies the following simplifications to the list results (compared to the inventory table): + **Column omissions** – The columns `bucket`, `is_multipart`, `encryption_status`, `is_bucket_key_enabled`, `kms_key_arn`, and `checksum_algorithm` aren't part of the final results. Keeping the set of optional columns to a minimum improves performance. + **Inclusion of all records** – The query returns all object keys and versions, including the null version (in unversioned or versioning-suspended buckets) and delete markers. For examples of how to filter the results to show only the keys that you're interested in, see the `WHERE` clause at the end of the query. + **Accelerated reconciliation** – The query could, in rare cases, temporarily report objects that are no longer in the bucket. Those discrepancies are eliminated as soon as the next snapshot of the inventory table becomes available. This behavior is a tradeoff between performance and accuracy. To run this query in Amazon Athena, make sure to select the `s3tablescatalog/aws-s3` catalog and the `b_general-purpose-bucket-name` database for the general purpose bucket metadata configuration that contains your journal and inventory tables. ``` WITH inventory_time_cte AS ( SELECT COALESCE(inventory_time_from_property, inventory_time_default) AS inventory_time FROM ( SELECT * FROM (VALUES (TIMESTAMP '2024-12-01 00:00')) AS T (inventory_time_default) LEFT OUTER JOIN ( SELECT from_unixtime(CAST(value AS BIGINT) / 1000.0) AS inventory_time_from_property FROM "journal$properties" WHERE key = 'aws.s3metadata.oldest-uncoalesced-record-timestamp' LIMIT 1 ) ON TRUE ) ), working_set AS ( SELECT key, sequence_number, version_id, is_delete_marker, size, COALESCE(last_modified_date, record_timestamp) AS last_modified_date, e_tag, storage_class, object_tags, user_metadata, (record_type = 'DELETE' AND NOT COALESCE(is_delete_marker, FALSE)) AS _is_perm_delete FROM journal j CROSS JOIN inventory_time_cte t WHERE j.record_timestamp > (t.inventory_time - interval '15' minute) UNION ALL SELECT key, sequence_number, version_id, is_delete_marker, size, last_modified_date, e_tag, storage_class, object_tags, user_metadata, FALSE AS _is_perm_delete FROM inventory i ), updated_inventory AS ( SELECT * FROM ( SELECT *, MAX(sequence_number) OVER (PARTITION BY key, version_id) AS _supremum_sn FROM working_set ) WHERE sequence_number = _supremum_sn ) SELECT key, sequence_number, version_id, is_delete_marker, size, last_modified_date, e_tag, storage_class, object_tags, user_metadata FROM updated_inventory -- This filter omits only permanent deletes from the results. Delete markers will still be shown. WHERE NOT _is_perm_delete -- You can add additional filters here. Examples: -- AND object_tags['department'] = 'billing' -- AND starts_with(key, 'reports/') ORDER BY key ASC, sequence_number DESC; ``` ### Finding the current versions of your objects The following query uses the inventory table to generate a new output table that shows which object versions are current. The output table is intentionally similar to an S3 Inventory report. The output table includes an `is_latest` field, which indicates if an object is the current version. The `is_latest` field is equivalent to the **IsLatest** field in an [S3 Inventory report](storage-inventory.md#storage-inventory-contents). This query works for general purpose buckets with [S3 Versioning](Versioning.md) in a versioning-enabled or versioning-suspended state. **Prerequisites** The query outputs the results to a new S3 table to support further queries and for higher performance versus outputting rows on screen. Therefore, before running this query, make sure you've met the following conditions. If you choose not to output the results to a new table, you can skip these steps. + You must have an existing customer-managed table bucket with an existing namespace as a place to output the new table. For more information, see [Creating a table bucket](s3-tables-buckets-create.md) and [Creating a namespace](s3-tables-namespace-create.md). + To query your new output table, you must set up an access method for querying it. For more information, see [Accessing table data](s3-tables-access.md). If you want to query the output table with AWS analytics services such as Amazon Athena, your customer-managed table bucket must be integrated with AWS analytics services. For more information, see [Amazon S3 Tables integration with AWS analytics services overview](s3-tables-integration-overview.md). To use this query, replace `amzn-s3-demo-table-bucket` with the name of the existing customer-managed table bucket where you want the new output table to be created. Replace *`existing_namespace`* with the name of the namespace where you want the output table to be created in your table bucket. Replace *`new_table`* with the name that you want to use for the output table. Make sure that the name of your output table follows the [table naming rules](s3-tables-buckets-naming.md#naming-rules-table). To run this query in Amazon Athena, make sure to select the `s3tablescatalog/aws-s3` catalog and the `b_general-purpose-bucket-name` database for the general purpose bucket metadata configuration that contains your inventory table. ``` -- If you don't want to output the results to a new table, remove the following two lines -- (everything before the WITH clause). CREATE TABLE "s3tablescatalog/amzn-s3-demo-table-bucket"."existing_namespace"."new_table" as ( WITH my_inventory AS ( SELECT bucket, key, version_id, sequence_number, is_delete_marker, size, last_modified_date, storage_class FROM inventory -- For prefix filtering, use a WHERE clause with % at the end. -- WHERE key LIKE 'prefix%' ), inventory_with_is_latest as ( SELECT *, ROW_NUMBER() OVER ( PARTITION BY key ORDER BY sequence_number DESC ) = 1 AS is_latest FROM my_inventory ) SELECT bucket, key, version_id, sequence_number, is_delete_marker, size, last_modified_date, storage_class, is_latest FROM inventory_with_is_latest -- If you want only the current version of each key, uncomment the following WHERE clause. -- WHERE is_latest = TRUE -- If you aren't outputting the results to a new table, remove the next line: ); ``` # Joining custom metadata with S3 metadata tables You can analyze data across your AWS managed metadata tables and customer (self-managed) metadata tables. By using a standard SQL `JOIN` operator, you can query data from these multiple sources. The following example SQL query finds matching records between an AWS managed journal table (`"journal"`) and a self-managed metadata table (`my_self_managed_metadata_table`). The query also filters information based on `CREATE` events, which indicate that a new object (or a new version of the object) was written to the bucket. (For more information, see the [S3 Metadata journal tables schema](metadata-tables-schema.md).) ``` SELECT * FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."journal" a JOIN "my_namespace"."my_self_managed_metadata_table" b ON a.bucket = b.bucket AND a.key = b.key AND a.version_id = b.version_id WHERE a.record_type = 'CREATE'; ``` The following example SQL query finds matching records between an AWS managed inventory table (`"inventory"`) and a self-managed metadata table (`my_self_managed_metadata_table`): ``` SELECT * FROM "s3tablescatalog/aws-s3"."b_general-purpose-bucket-name"."inventory" a JOIN "my_namespace"."my_self_managed_metadata_table" b ON a.bucket = b.bucket AND a.key = b.key AND a.version_id = b.version_id; ``` # Visualizing metadata table data with Amazon Quick With Amazon Quick, you can create interactive dashboards to analyze and visualize SQL query results about your S3 managed metadata tables. Quick dashboards can help you monitor statistics, track changes, and get operational insights about your metadata tables. A dashboard about your journal table might show you: + What's the percentage of object uploads compared to deletions? + Which objects were deleted by S3 Lifecycle in the past 24 hours? + Which IP addresses did the most recent `PUT` requests come from? A dashboard about your inventory table might show you: + How many objects are in different storage classes? + What percentage of your storage data is small objects compared to large objects? + What types of objects are in my bucket? After you [integrate your S3 table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html) with AWS analytics services, you can create datasets from your metadata tables and work with them in Amazon Quick using SPICE or direct SQL queries from your query engine. Quick supports Amazon Athena and Amazon Redshift as data sources. For more information, see [Visualizing table data with Amazon Quick](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-quicksight.html). # Troubleshooting S3 Metadata Use the following information to help you diagnose and fix common issues that you might encounter when working with Amazon S3 Metadata. ## I'm unable to delete my AWS managed table bucket and metadata tables Before you can delete a metadata table, you must first delete the associated metadata table configuration on your general purpose bucket. For more information, see [Deleting metadata table configurations](metadata-tables-delete-configuration.md). Before you can delete your AWS managed table bucket, you must delete all metadata table configurations that are associated with this bucket and all metadata tables in the bucket. For more information, see [Deleting metadata table configurations](metadata-tables-delete-configuration.md) and [Deleting metadata tables](metadata-tables-delete-table.md). ## I'm unable to set or change the encryption settings for my AWS managed metadata table When you create your metadata table configuration, you can choose to encrypt your AWS managed metadata tables with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. You can set the encryption type for your tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. To specify SSE-KMS for your metadata tables, you must have certain permissions. For more information, see [ Permissions for SSE-KMS](metadata-tables-permissions.md#metadata-kms-permissions). The encryption setting for a metadata table takes precedence over the default bucket-level encryption setting. If you don't specify encryption for a table, it will inherit the default encryption setting from the bucket. By default, AWS managed table buckets are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). After you create your first metadata configuration, you can set the default encryption setting for the AWS managed table bucket to use server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). For more information, see [Encryption for AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html#aws-managed-buckets-encryption) and [Specifying server-side encryption with AWS KMS keys (SSE-KMS) in table buckets](s3-tables-kms-specify.md). ## When I try to re-create my metadata table configuration, I get an error Deleting a metadata table configuration deletes only the configuration. The AWS managed table bucket and your metadata tables still exist, even if you delete the metadata table configuration. If you delete your metadata table configuration and want to re-create a configuration for the same general purpose bucket, you must first manually delete the old journal and inventory tables from your AWS managed table bucket. Otherwise, creating the new metadata table configuration fails because those tables already exist. To delete your metadata tables, see [Deleting metadata tables](metadata-tables-delete-table.md). ## I can't enable an inventory table on my configuration If you created your S3 Metadata configuration before July 15, 2025, you can't enable an inventory table on that configuration. We recommend that you delete and re-create your configuration so that you can create an inventory table and expire journal table records. For more information, see [Enabling inventory tables on metadata configurations created before July 15, 2025](metadata-tables-create-configuration.md#metadata-tables-migration). ## I can't enable journal table record expiration on my configuration If you created your S3 Metadata configuration before July 15, 2025, you can't enable journal table record expiration on that configuration. We recommend that you delete and re-create your configuration so that you can expire journal table records and create an inventory table. For more information, see [Enabling inventory tables on metadata configurations created before July 15, 2025](metadata-tables-create-configuration.md#metadata-tables-migration). ## I can't query my metadata tables If you're unable to query your metadata tables, check the following: + When you're using Amazon Athena or Amazon Redshift to query your metadata tables, you must surround your metadata table namespace names in quotation marks (`"`) or backticks (```), otherwise the query might not work. + When using Apache Spark on Amazon EMR or other third-party engines to query your metadata tables, we recommend that you use the Amazon S3 Tables Iceberg REST endpoint. Your query might not run successfully if you don't use this endpoint. For more information, see [Accessing tables using the Amazon S3 Tables Iceberg REST endpoint](s3-tables-integrating-open-source.md). + Make sure that you have the appropriate AWS Identity and Access Management (IAM) permissions to query metadata tables. For more information, see [Permissions for querying metadata tables](metadata-tables-bucket-query-permissions.md). + If you're using Amazon Athena and receive errors when you try to run your queries, do the following: + If you receive the error "Insufficient permissions to execute the query. Principal does not have any privilege on specified resource" when you try to run a query in Athena, you must be granted the necessary Lake Formation permissions on the table. For more information, see [Granting Lake Formation permission on a table or database](grant-permissions-tables.md#grant-lf-table). + If you receive the error "Iceberg cannot access the requested resource" when you try to run the query, go to the AWS Lake Formation console and make sure that you've granted yourself permissions on the table bucket catalog and database (namespace) that you created. Don't specify a table when granting these permissions. For more information, see [Granting Lake Formation permission on a table or database](grant-permissions-tables.md#grant-lf-table). ## I'm receiving 405 errors when I try to use certain S3 Metadata AWS CLI commands and API operations Calling the V1 `GetBucketMetadataTableConfiguration` API operation or using the `get-bucket-metadata-table-configuration` AWS Command Line Interface (AWS CLI) command against a V2 metadata table configuration results in an HTTP `405 Method Not Allowed` error. Likewise, calling the V1 `DeleteBucketMetadataTableConfiguration` API operation or using the `delete-bucket-metadata-table-configuration` AWS CLI command also causes a 405 error. You can use the V2 `GetBucketMetadataConfiguration` API operation or the `get-bucket-metadata-configuration` AWS CLI command against a V1 or V2 metadata table configuration. Likewise, you can use the V2 `DeleteBucketMetadataConfiguration` API operation or the `delete-bucket-metadata-configuration` AWS CLI command against a V1 or V2 metadata table configuration. We recommend updating your processes to use the new V2 API operations (`CreateBucketMetadataConfiguration`, `GetBucketMetadataConfiguraion`, and `DeleteBucketMetadataConfiguration`) instead of the V1 API operations. For more information about migrating from V1 of S3 Metadata to V2, see [Enabling inventory tables on metadata configurations created before July 15, 2025](metadata-tables-create-configuration.md#metadata-tables-migration). To determine whether your configuration is V1 or V2, you can look at the following attribute of your `GetBucketMetadataConfiguration` API response. An AWS managed bucket type (`"aws"`) indicates a V2 configuration, and a customer-managed bucket type (`"customer"`) indicates a V1 configuration. ``` "MetadataTableConfigurationResult": { "TableBucketType": ["aws" | "customer"] ``` For more information, see [Viewing metadata table configurations](metadata-tables-view-configuration.md). # Uploading objects When you upload a file to Amazon S3, it is stored as an S3 *object*. Objects consist of the file data and metadata that describes the object. You can have an unlimited number of objects in a bucket. Before you can upload files to an Amazon S3 bucket, you need write permissions for the bucket. For more information about access permissions, see [Identity and Access Management for Amazon S3](security-iam.md). You can upload any file type—images, backups, data, movies, and so on—into an S3 bucket. The maximum size of a file that you can upload by using the Amazon S3 console is 160 GB. To upload a file larger than 160 GB, use the AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. If you upload an object with a key name that already exists in a versioning-enabled bucket, Amazon S3 creates another version of the object instead of replacing the existing object. For more information about enabling versioning, see [Enabling versioning on buckets](manage-versioning-examples.md). Depending on the size of the data that you're uploading, Amazon S3 offers the following options: + **Upload an object in a single operation by using the AWS SDKs, REST API, or AWS CLI** – With a single `PUT` operation, you can upload a single object up to 5 GB in size. + **Upload a single object by using the Amazon S3 console**** –** With the Amazon S3 console, you can upload a single object up to 160 GB in size. + **Upload an object in parts by using the AWS SDKs, REST API, or AWS CLI**** –** Using the multipart upload API operation, you can upload a single large object, up to 50 TB in size. The multipart upload API operation is designed to improve the upload experience for larger objects. You can upload an object in parts. These object parts can be uploaded independently, in any order, and in parallel. You can use a multipart upload for objects from 5 MB to 50 TB in size. For more information, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). To upload files greater than 5 TB, use the S3 Transfer Manager in the Java v1/v2, Python, or AWS CLI SDKs. For the best performance, use the latest AWS Common Runtime (CRT) with these SDKs, which has been optimized for better resource utilization. When uploading large objects from memory stream, CRT buffers each part up to 5 GB in memory, limiting overall throughput by allocated memory. You can adjust the CRT memory limit using configuration options such as `maxNativeMemoryLimitInBytes` for Java SDK. For uploads from disk, CRT automatically switches to direct disk streaming instead of intermediate part buffering, improving memory usage. This behavior is automatically enabled for large objects, but can also be enabled for smaller files via request parameters such as `should_stream` for AWS CLI and `CRT_MEMORY_BUFFER_DISABLED` for Java SDK. When you upload an object, the object is automatically encrypted using server-side encryption with Amazon S3 managed keys (SSE-S3) by default. When you download it, the object is decrypted. For more information, see [Setting default server-side encryption behavior for Amazon S3 buckets](bucket-encryption.md) and [Protecting data with encryption](UsingEncryption.md). When you're uploading an object, if you want to use a different type of default encryption, you can also specify server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS) in your S3 `PUT` requests or set the default encryption configuration in the destination bucket to use SSE-KMS to encrypt your data. For more information about SSE-KMS, see [Specifying server-side encryption with AWS KMS (SSE-KMS)](specifying-kms-encryption.md). If you want to use a KMS key that is owned by a different account, you must have permission to use the key. For more information about cross-account permissions for KMS keys, see [Creating KMS keys that other accounts can use](https://docs.aws.amazon.com//kms/latest/developerguide/key-policy-modifying-external-accounts.html#cross-account-console) in the *AWS Key Management Service Developer Guide*. If you encounter an Access Denied (403 Forbidden) error in Amazon S3, see [Troubleshoot access denied (403 Forbidden) errors in Amazon S3](troubleshoot-403-errors.md) to learn more about its common causes. ## Upload an object ### Using the S3 console This procedure explains how to upload objects and folders to an Amazon S3 bucket by using the console. When you upload an object, the object key name is the file name and any optional prefixes. In the Amazon S3 console, you can create folders to organize your objects. In Amazon S3, folders are represented as prefixes that appear in the object key name. If you upload an individual object to a folder in the Amazon S3 console, the folder name is included in the object key name. For example, if you upload an object named `sample1.jpg` to a folder named `backup`, the key name is `backup/sample1.jpg`. However, the object is displayed in the console as `sample1.jpg` in the `backup` folder. For more information about key names, see [Working with object metadata](UsingMetadata.md). **Note** If you rename an object or change any of the properties in the Amazon S3 console, for example **Storage Class**, ** Encryption**, or **Metadata**, a new object is created to replace the old one. If S3 Versioning is enabled, a new version of the object is created, and the existing object becomes an older version. The role that changes the property also becomes the owner of the new object (or object version). When you upload a folder, Amazon S3 uploads all of the files and subfolders from the specified folder to your bucket. It then assigns an object key name that is a combination of the uploaded file name and the folder name. For example, if you upload a folder named `/images` that contains two files, `sample1.jpg` and `sample2.jpg`, Amazon S3 uploads the files and then assigns the corresponding key names, `images/sample1.jpg` and `images/sample2.jpg`. The key names include the folder name as a prefix. The Amazon S3 console displays only the part of the key name that follows the last `/`. For example, within an `images` folder, the `images/sample1.jpg` and `images/sample2.jpg` objects are displayed as `sample1.jpg` and a `sample2.jpg`. **To upload folders and files to an S3 bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the name of the bucket that you want to upload your folders or files to. 1. Choose **Upload**. 1. In the **Upload** window, do one of the following: + Drag and drop files and folders to the **Upload** window. + Choose **Add file** or **Add folder**, choose the files or folders to upload, and choose **Open**. 1. To enable versioning, under **Destination**, choose **Enable Bucket Versioning**. 1. To upload the listed files and folders without configuring additional upload options, at the bottom of the page, choose **Upload**. Amazon S3 uploads your objects and folders. When the upload is finished, you see a success message on the **Upload: status** page. **To configure additional object properties** 1. To change access control list permissions, choose **Permissions**. 1. Under **Access control list (ACL)**, edit the permissions. For information about object access permissions, see [Using the S3 console to set ACL permissions for an object](managing-acls.md#set-object-permissions). You can grant read access to your objects to the public (everyone in the world) for all of the files that you're uploading. However, we recommend not changing the default setting for public read access. Granting public read access is applicable to a small subset of use cases, such as when buckets are used for websites. You can always change the object permissions after you upload the object. 1. To configure other additional properties, choose **Properties**. 1. Under **Storage class**, choose the storage class for the files that you're uploading. For more information about storage classes, see [Understanding and managing Amazon S3 storage classes](storage-class-intro.md). 1. To update the encryption settings for your objects, under **Server-side encryption settings**, do the following. 1. Choose **Specify an encryption key**. 1. Under **Encryption settings**, choose **Use bucket settings for default encryption** or **Override bucket settings for default encryption**. 1. If you chose **Override bucket settings for default encryption**, you must configure the following encryption settings. + To encrypt the uploaded files by using keys that are managed by Amazon S3, choose **Amazon S3 managed key (SSE-S3)**. For more information, see [Using server-side encryption with Amazon S3 managed keys (SSE-S3)](UsingServerSideEncryption.md). + To encrypt the uploaded files by using keys stored in AWS Key Management Service (AWS KMS), choose **AWS Key Management Service key (SSE-KMS)**. Then choose one of the following options for **AWS KMS key**: + To choose from a list of available KMS keys, choose **Choose from your AWS KMS keys**, and then choose your **KMS key** from the list of available keys. Both the AWS managed key (`aws/s3`) and your customer managed keys appear in this list. For more information about customer managed keys, see [Customer keys and AWS keys](https://docs.aws.amazon.com//kms/latest/developerguide/concepts.html#key-mgmt) in the *AWS Key Management Service Developer Guide*. + To enter the KMS key ARN, choose **Enter AWS KMS key ARN**, and then enter your KMS key ARN in the field that appears. + To create a new customer managed key in the AWS KMS console, choose **Create a KMS key**. For more information about creating an AWS KMS key, see [Creating keys](https://docs.aws.amazon.com//kms/latest/developerguide/create-keys.html) in the *AWS Key Management Service Developer Guide*. **Important** You can use only KMS keys that are available in the same AWS Region as the bucket. The Amazon S3 console lists only the first 100 KMS keys in the same Region as the bucket. To use a KMS key that is not listed, you must enter your KMS key ARN. If you want to use a KMS key that is owned by a different account, you must first have permission to use the key and then you must enter the KMS key ARN. Amazon S3 supports only symmetric encryption KMS keys, and not asymmetric KMS keys. For more information, see [Identifying symmetric and asymmetric KMS keys](https://docs.aws.amazon.com//kms/latest/developerguide/find-symm-asymm.html) in the *AWS Key Management Service Developer Guide*. 1. To use additional checksums, choose **On**. Then for **Checksum function**, choose the function that you would like to use. Amazon S3 calculates and stores the checksum value after it receives the entire object. You can use the **Precalculated value** box to supply a precalculated value. If you do, Amazon S3 compares the value that you provided to the value that it calculates. If the two values do not match, Amazon S3 generates an error. Additional checksums enable you to specify the checksum algorithm that you would like to use to verify your data. For more information about additional checksums, see [Checking object integrity in Amazon S3](checking-object-integrity.md). 1. To add tags to all of the objects that you are uploading, choose **Add tag**. Enter a tag name in the **Key** field. Enter a value for the tag. Object tagging gives you a way to categorize storage. Each tag is a key-value pair. Key and tag values are case sensitive. You can have up to 10 tags per object. A tag key can be up to 128 Unicode characters in length, and tag values can be up to 255 Unicode characters in length. For more information about object tags, see [Categorizing your objects using tags](object-tagging.md). 1. To add metadata, choose **Add metadata**. 1. Under **Type**, choose **System defined** or **User defined**. For system-defined metadata, you can select common HTTP headers, such as **Content-Type** and **Content-Disposition**. For a list of system-defined metadata and information about whether you can add the value, see [System-defined object metadata](UsingMetadata.md#SysMetadata). Any metadata starting with the prefix `x-amz-meta-` is treated as user-defined metadata. User-defined metadata is stored with the object and is returned when you download the object. Both the keys and their values must conform to US-ASCII standards. User-defined metadata can be as large as 2 KB. For more information about system-defined and user-defined metadata, see [Working with object metadata](UsingMetadata.md). 1. For **Key**, choose a key. 1. Type a value for the key. 1. To upload your objects, choose **Upload**. Amazon S3 uploads your object. When the upload completes, you can see a success message on the **Upload: status** page. 1. Choose **Exit**. ### Using the AWS CLI You can send a `PUT` request to upload an object of up to 5 GB in a single operation. For more information, see the [https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html#examples](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html#examples) example in the *AWS CLI Command Reference*. ### Using the REST API You can send REST requests to upload an object. You can send a `PUT` request to upload data in a single operation. For more information, see [PUT Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html). ### Using the AWS SDKs For examples of how to upload an object with the AWS SDKs, see [Code Examples](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_PutObject_section.html) in the *Amazon Simple Storage Service API Reference*. For general information about using different AWS SDKs, see [Developing with Amazon S3 using the AWS SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/sdk-general-information-section.html) in the *Amazon Simple Storage Service API Reference*. ## Prevent uploading objects with identical key names You can check for the existence of an object in your bucket before creating it using a conditional write on upload operations. This can prevent overwrites of existing data. Conditional writes will validate there is no existing object with the same key name already in your bucket while uploading. You can use conditional writes for [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) or [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) requests. For more information about conditional requests see, [Add preconditions to S3 operations with conditional requests](conditional-requests.md). # Uploading and copying objects using multipart upload in Amazon S3 Multipart upload allows you to upload a single object to Amazon S3 as a set of parts. Each part is a contiguous portion of the object's data. You can upload these object parts independently, and in any order. For uploads, your updated AWS client automatically calculates a checksum of the object and sends it to Amazon S3 along with the size of the object as a part of the request. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles them to create the object. It's a best practice to use multipart upload for objects that are 100 MB or larger instead of uploading them in a single operation. Using multipart upload provides the following advantages: + **Improved throughput** – You can upload parts in parallel to improve throughput. + **Quick recovery from any network issues** – Smaller part size minimizes the impact of restarting a failed upload due to a network error. + **Pause and resume object uploads** – You can upload object parts over time. After you initiate a multipart upload, there is no expiry; you must explicitly complete or stop the multipart upload. + **Begin an upload before you know the final object size** – You can upload an object as you create it. We recommend that you use multipart upload in the following ways: + If you upload large objects over a stable high-bandwidth network, use multipart upload to maximize the use of your available bandwidth by uploading object parts in parallel for multi-threaded performance. + If you upload over a spotty network, use multipart upload to increase resiliency against network errors by avoiding upload restarts. When using multipart upload, you only need to retry uploading the parts that are interrupted during the upload. You don't need to restart uploading your object from the beginning. **Note** For more information about using the Amazon S3 Express One Zone storage class with directory buckets, see [S3 Express One Zone](directory-bucket-high-performance.md#s3-express-one-zone) and [Working with directory buckets](directory-buckets-overview.md). For more information about using multipart upload with S3 Express One Zone and directory buckets, see [Using multipart uploads with directory buckets](s3-express-using-multipart-upload.md). ## Multipart upload process Multipart upload is a three-step process: You initiate the upload, upload the object parts, and—after you've uploaded all the parts—complete the multipart upload. Upon receiving the complete multipart upload request, Amazon S3 constructs the object from the uploaded parts, and you can access the object just as you would any other object in your bucket. You can list all of your in-progress multipart uploads or get a list of the parts that you have uploaded for a specific multipart upload. Each of these operations is explained in this section. **Multipart upload initiation** When you send a request to initiate a multipart upload, make sure to specify a checksum type. Amazon S3 will then return a response with an upload ID, which is a unique identifier for your multipart upload. This upload ID is required when you upload parts, list parts, complete an upload, or stop an upload. If you want to provide metadata describing the object being uploaded, you must provide it in the request to initiate the multipart upload. Anonymous users cannot initiate multipart uploads. **Parts upload** When uploading a part, you must specify a part number in addition to the upload ID. You can choose any part number between 1 and 10,000. A part number uniquely identifies a part and its position in the object you are uploading. The part number that you choose doesn’t need to be in a consecutive sequence (for example, it can be 1, 5, and 14). Be aware that if you upload a new part using the same part number as a previously uploaded part, the previously uploaded part gets overwritten. When you upload a part, Amazon S3 returns the checksum algorithm type with the checksum value for each part as a header in the response. For each part upload, you must record the part number and the ETag value. You must include these values in the subsequent request to complete the multipart upload. Each part will have its own ETag at the time of upload. However, once the multipart upload is complete and all parts are consolidated, all parts belong to one ETag as a checksum of checksums. **Important** After you initiate a multipart upload and upload one or more parts, you must either complete or stop the multipart upload to stop incurring charges for storage of the uploaded parts. Only *after* you complete or stop a multipart upload will Amazon S3 free up the parts storage and stop billing you for the parts storage. After stopping a multipart upload, you can't upload any part using that upload ID again. If part uploads were in progress, they can still succeed or fail even after you stop the upload. To make sure you free all storage consumed by all parts, you must stop a multipart upload only after all part uploads have completed. **Multipart upload completion** When you complete a multipart upload, Amazon S3 creates an object by concatenating the parts in ascending order based on the part number. If any object metadata was provided in the *initiate multipart upload* request, Amazon S3 associates that metadata with the object. After a successful *complete* request, the parts no longer exist. Your *complete multipart upload* request must include the upload ID and a list of part numbers and their corresponding ETag values. The Amazon S3 response includes an ETag that uniquely identifies the combined object data. This ETag is not necessarily an MD5 hash of the object data. When you provide a full object checksum during a multipart upload, the AWS SDK passes the checksum to Amazon S3, and S3 validates the object integrity server-side, comparing it to the received value. Then, S3 stores the object if the values match. If the two values don’t match, Amazon S3 fails the request with a `BadDigest` error. The checksum of your object is also stored in object metadata that you'll later use to validate an object's data integrity. **Sample multipart upload calls** For this example, assume that you're generating a multipart upload for a 100 GB file. In this case, you would have the following API calls for the entire process. There would be a total of 1,002 API calls. + A `[CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html)` call to start the process. + 1,000 individual `[UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html)` calls, each uploading a part of 100 MB, for a total size of 100 GB. + A `[CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html)` call to complete the process. **Multipart upload listings** You can list the parts of a specific multipart upload or all in-progress multipart uploads. The list parts operation returns the parts information that you uploaded for a specific multipart upload. For each list parts request, Amazon S3 returns the parts information for the specified multipart upload, up to a maximum of 1,000 parts. If there are more than 1,000 parts in the multipart upload, you must send a series of list part requests to retrieve all of the parts. Note that the returned list of parts doesn't include parts that haven't finished uploading. Using the *list multipart uploads* operation, you can obtain a list of multipart uploads that are in progress. An in-progress multipart upload is an upload that you have initiated, but have not yet completed or stopped. Each request returns at most 1,000 multipart uploads. If there are more than 1,000 multipart uploads in progress, you must send additional requests to retrieve the remaining multipart uploads. Use the returned listing only for verification. **Important** Do not use the result of this listing when sending a *complete multipart upload* request. Instead, maintain your own list of the part numbers that you specified when uploading parts and the corresponding ETag values that Amazon S3 returns. ## Checksums with multipart upload operations When you upload an object to Amazon S3, you can specify a checksum algorithm for Amazon S3 to use. By default, the AWS SDK and S3 console use an algorithm for all object uploads, which you can override. If you’re using an older SDK and your uploaded object doesn’t have a specified checksum, Amazon S3 automatically uses the CRC-64/NVME (`CRC64NVME`) checksum algorithm. (This is also the recommended option for efficient data integrity verification.) When using CRC-64/NVME, Amazon S3 calculates the checksum of the full object after the multipart or single part upload is complete. The CRC-64/NVME checksum algorithm is used to calculate either a direct checksum of the entire object, or a checksum of the checksums, for each individual part. After you upload an object to S3 using multipart upload, Amazon S3 calculates the checksum value for each part, or for the full object—and stores the values. You can use the S3 API or AWS SDK to retrieve the checksum value in the following ways: + For individual parts, you can use [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) or [https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). If you want to retrieve the checksum values for individual parts of multipart uploads while they're still in process, you can use [https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html). + For the entire object, you can use [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html). If you want to perform a multipart upload with a full object checksum, use [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload) and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload) by specifying the full object checksum type. To validate the checksum value of the entire object or to confirm which checksum type is being used in the multipart upload, use [https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html). **Important** If you're using a multipart upload with **Checksums**, the part numbers for each part upload (in the multipart upload) must use consecutive part numbers and begin with 1. When using **Checksums**, if you try to complete a multipart upload request with nonconsecutive part numbers, Amazon S3 generates an `HTTP 500 Internal Server` error. For more information about how checksums work with multipart upload objects, see [Checking object integrity in Amazon S3](checking-object-integrity.md). For an end-to-end procedure that demonstrates how to upload an object using multipart upload with an additional checksum, see [Tutorial: Upload an object through multipart upload and verify its data integrity](tutorial-s3-mpu-additional-checksums.md). ## Concurrent multipart upload operations In a distributed development environment, it is possible for your application to initiate several updates on the same object at the same time. Your application might initiate several multipart uploads using the same object key. For each of these uploads, your application can then upload parts and send a complete upload request to Amazon S3 to create the object. When the buckets have S3 Versioning enabled, completing a multipart upload always creates a new version. When you initiate multiple multipart uploads that use the same object key in a versioning-enabled bucket, the current version of the object is determined by which upload started most recently (`createdDate`). For example, you start a `CreateMultipartUpload` request for an object at 10:00 AM. Then, you submit a second `CreateMultipartUpload` request for the same object at 11:00 AM. Because the second request was submitted the most recently, the object uploaded by the 11:00 AM request becomes the current version, even if the first upload is completed after the second one. For buckets that don't have versioning enabled, it's possible that any other request received between the time when the multipart upload is initiated and when it completes, the other request might take precedence. Another example of when a concurrent multipart upload request can take precedence is if another operation deletes a key after you initiate a multipart upload with that key. Before you complete the operation, the complete multipart upload response might indicate a successful object creation without you ever seeing the object. ## Prevent uploading objects with identical key names during multipart upload You can check for the existence of an object in your bucket before creating it using a conditional write on upload operations. This can prevent overwrites of existing data. Conditional writes will validate that there is no existing object with the same key name already in your bucket while uploading. You can use conditional writes for [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) or [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) requests. For more information about conditional requests see, [Add preconditions to S3 operations with conditional requests](conditional-requests.md). ## Multipart upload and pricing After you initiate a multipart upload, Amazon S3 retains all the parts until you either complete or stop the upload. Throughout its lifetime, you are billed for all storage, bandwidth, and requests for this multipart upload and its associated parts. These parts are billed according to the storage class specified when the parts are uploaded. However, you will not be billed for these parts if they're uploaded to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive. In-progress multipart parts for a PUT request to the S3 Glacier Flexible Retrieval storage class are billed as S3 Glacier Flexible Retrieval staging storage at S3 Standard storage rates until the upload completes. In addition, both `CreateMultipartUpload` and `UploadPart` are billed at S3 Standard rates. Only the `CompleteMultipartUpload` request is billed at the S3 Glacier Flexible Retrieval rate. Similarly, in-progress multipart parts for a PUT to the S3 Glacier Deep Archive storage class are billed as S3 Glacier Flexible Retrieval staging storage at S3 Standard storage rates until the upload completes, with only the `CompleteMultipartUpload` request charged at S3 Glacier Deep Archive rates. If you stop the multipart upload, Amazon S3 deletes upload artifacts and all parts that you uploaded. You will not be billed for those artifacts. There are no early delete charges for deleting incomplete multipart uploads regardless of storage class specified. For more information about pricing, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/). **Note** To minimize your storage costs, we recommend that you configure a lifecycle rule to delete incomplete multipart uploads after a specified number of days by using the `AbortIncompleteMultipartUpload` action. For more information about creating a lifecycle rule to delete incomplete multipart uploads, see [Configuring a bucket lifecycle configuration to delete incomplete multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html). ## API support for multipart upload The following sections in the *Amazon Simple Storage Service API Reference* describe the REST API for multipart upload. For a multipart upload walkthrough that uses AWS Lambda functions, see [Uploading large objects to Amazon S3 using multipart upload and transfer acceleration](https://aws.amazon.com/blogs/compute/uploading-large-objects-to-amazon-s3-using-multipart-upload-and-transfer-acceleration/). + [Create Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) + [Upload Part](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) + [Upload Part (Copy)](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html) + [Complete Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) + [Abort Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) + [List Parts](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html) + [List Multipart Uploads](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html) ## AWS Command Line Interface support for multipart upload The following topics in the AWS Command Line Interface describe the operations for multipart upload. + [Initiate Multipart Upload](https://docs.aws.amazon.com/cli/latest/reference/s3api/create-multipart-upload.html) + [Upload Part](https://docs.aws.amazon.com/cli/latest/reference/s3api/upload-part.html) + [Upload Part (Copy)](https://docs.aws.amazon.com/cli/latest/reference/s3api/upload-part-copy.html) + [Complete Multipart Upload](https://docs.aws.amazon.com/cli/latest/reference/s3api/complete-multipart-upload.html) + [Abort Multipart Upload](https://docs.aws.amazon.com/cli/latest/reference/s3api/abort-multipart-upload.html) + [List Parts](https://docs.aws.amazon.com/cli/latest/reference/s3api/list-parts.html) + [List Multipart Uploads](https://docs.aws.amazon.com/cli/latest/reference/s3api/list-multipart-uploads.html) ## AWS SDK support for multipart upload You can use an AWS SDKs to upload an object in parts. For a list of AWS SDKs supported by API action see: + [Create Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) + [Upload Part](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) + [Upload Part (Copy)](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html) + [Complete Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) + [Abort Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) + [List Parts](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html) + [List Multipart Uploads](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html) ## Multipart upload API and permissions You must have the necessary permissions to use the multipart upload operations. You can use access control lists (ACLs), the bucket policy, or the user policy to grant individuals permissions to perform these operations. The following table lists the required permissions for various multipart upload operations when using ACLs, a bucket policy, or a user policy. | Action | Required permissions | | --- | --- | | Create Multipart Upload | You must be allowed to perform the `s3:PutObject` action on an object to create a multipart upload request. The bucket owner can allow other principals to perform the `s3:PutObject` action. | | Initiate Multipart Upload | You must be allowed to perform the `s3:PutObject` action on an object to initiate a multipart upload. The bucket owner can allow other principals to perform the `s3:PutObject` action. | | Initiator | Container element that identifies who initiated the multipart upload. If the initiator is an AWS account, this element provides the same information as the Owner element. If the initiator is an IAM user, this element provides the user ARN and display name. | | Upload Part | You must be allowed to perform the `s3:PutObject` action on an object to upload a part. The bucket owner must allow the initiator to perform the `s3:PutObject` action on an object in order for the initiator to upload a part for that object. | | Upload Part (Copy) | You must be allowed to perform the `s3:PutObject` action on an object to upload a part. Because you are uploading a part from an existing object, you must be allowed `s3:GetObject` on the source object. For the initiator to upload a part for an object, the owner of the bucket must allow the initiator to perform the `s3:PutObject` action on the object. | | Complete Multipart Upload | You must be allowed to perform the `s3:PutObject` action on an object to complete a multipart upload. The bucket owner must allow the initiator to perform the `s3:PutObject` action on an object in order for the initiator to complete a multipart upload for that object. | | Stop Multipart Upload | You must be allowed to perform the `s3:AbortMultipartUpload` action to stop a multipart upload. By default, the bucket owner and the initiator of the multipart upload are allowed to perform this action as a part of IAM and S3 bucket polices. If the initiator is an IAM user, that user's AWS account is also allowed to stop that multipart upload. With VPC endpoint policies, the initiator of the multipart upload doesn't automatically gain the permission to perform the `s3:AbortMultipartUpload` action. In addition to these defaults, the bucket owner can allow other principals to perform the `s3:AbortMultipartUpload` action on an object. The bucket owner can deny any principal the ability to perform the `s3:AbortMultipartUpload` action. | | List Parts | You must be allowed to perform the `s3:ListMultipartUploadParts` action to list parts in a multipart upload. By default, the bucket owner has permission to list parts for any multipart upload to the bucket. The initiator of the multipart upload has the permission to list parts of the specific multipart upload. If the multipart upload initiator is an IAM user, the AWS account controlling that IAM user also has permission to list parts of that upload. In addition to these defaults, the bucket owner can allow other principals to perform the `s3:ListMultipartUploadParts` action on an object. The bucket owner can also deny any principal the ability to perform the `s3:ListMultipartUploadParts` action. | | List Multipart Uploads | You must be allowed to perform the `s3:ListBucketMultipartUploads` action on a bucket to list multipart uploads in progress to that bucket. In addition to the default, the bucket owner can allow other principals to perform the `s3:ListBucketMultipartUploads` action on the bucket. | | AWS KMS Encrypt and Decrypt related permissions | To perform a multipart upload with encryption using an AWS Key Management Service (AWS KMS) KMS key, the requester must have the following permissions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) These permissions are required because Amazon S3 must decrypt and read data from the encrypted file parts before it completes the multipart upload. The `kms:Decrypt` permission, and the server-side encryption with customer-provided encryption keys, are also required for you to obtain an object's checksum value. If you don't have these required permissions when you use the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) API, the object is created without a checksum value. If your IAM user or role is in the same AWS account as the KMS key, then validate that you have permissions on both the key and IAM policies. If your IAM user or role belongs to a different account than the KMS key, then you must have the permissions on both the key policy and your IAM user or role. | | SSE-C (server-side encryption with customer-provided encryption keys) | When you use the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) API, you must provide the SSE-C (server-side encryption with customer-provided encryption keys), or your object will be created without a checksum, and no checksum value is returned. | For information on the relationship between ACL permissions and permissions in access policies, see [Mapping of ACL permissions and access policy permissions](acl-overview.md#acl-access-policy-permission-mapping). For information about IAM users, roles, and best practices, see [IAM identities (users, user groups, and roles)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html) in the *IAM User Guide*. ## Checksums with multipart upload operations There are three Amazon S3 APIs that are used to perform the actual multipart upload: [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html), [https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html), and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html). The following table indicates which checksum headers and values must be provided for each of the APIs: | Checksum algorithm | Checksum type | `CreateMultipartUpload` | `UploadPart` | `CompleteMultipartUpoad` | | --- | --- | --- | --- | --- | | CRC-64/NVME (`CRC64NVME`) | Full object | Required headers: `x-amz-checksum-algorithm` | Optional headers: `x-amz-checksum-crc64nvme` | Optional headers: `x-amz-checksum-algorithm` `x-amz-crc64` | | CRC-32 (`CRC32`) CRC 32-C (`CRC32C`) | Full object | Required headers: `x-amz-checksum-algorithm` `x-amz-checksum-type` | Optional headers: `x-amz-checksum-crc64nvme` | Optional headers: `x-amz-checksum-algorithm` `x-amz-crc32` `x-amz-crc32c` | | CRC-32 (`CRC32`) CRC-32C (`CRC32C`) SHA-1 (`SHA1`) SHA-256 (`SHA256`) | Composite | Required headers: `x-amz-checksum-algorithm` | Required headers: `x-amz-checksum-crc32` `x-amz-checksum-crc32c` `x-amz-checksum-sha1` `x-amz-checksum-sha256` | Required headers: All part-level checksums need to be included in the `CompleteMultiPartUpload` request. Optional headers: `x-amz-crc32` `x-amz-crc32c` `x-amz-sha1` `x-amz-sha256` | **Topics** + [ ## Multipart upload process ](#mpu-process) + [ ## Checksums with multipart upload operations ](#mpuchecksums) + [ ## Concurrent multipart upload operations ](#distributedmpupload) + [ ## Prevent uploading objects with identical key names during multipart upload ](#multipart-upload-objects-with-same-key-name) + [ ## Multipart upload and pricing ](#mpuploadpricing) + [ ## API support for multipart upload ](#apisupportformpu) + [ ## AWS Command Line Interface support for multipart upload ](#clisupportformpu) + [ ## AWS SDK support for multipart upload ](#sdksupportformpu) + [ ## Multipart upload API and permissions ](#mpuAndPermissions) + [ ## Checksums with multipart upload operations ](#Checksums-mpu-operations) + [ # Configuring a bucket lifecycle configuration to delete incomplete multipart uploads ](mpu-abort-incomplete-mpu-lifecycle-config.md) + [ # Uploading an object using multipart upload ](mpu-upload-object.md) + [ # Uploading a directory using the high-level .NET TransferUtility class ](HLuploadDirDotNet.md) + [ # Listing multipart uploads ](list-mpu.md) + [ # Tracking a multipart upload with the AWS SDKs ](track-mpu.md) + [ # Aborting a multipart upload ](abort-mpu.md) + [ # Copying an object using multipart upload ](CopyingObjectsMPUapi.md) + [ # Tutorial: Upload an object through multipart upload and verify its data integrity ](tutorial-s3-mpu-additional-checksums.md) + [ # Amazon S3 multipart upload limits ](qfacts.md) # Configuring a bucket lifecycle configuration to delete incomplete multipart uploads As a best practice, we recommend that you configure a lifecycle rule by using the `AbortIncompleteMultipartUpload` action to minimize your storage costs. For more information about aborting a multipart upload, see [Aborting a multipart upload](abort-mpu.md). Amazon S3 supports a bucket lifecycle rule that you can use to direct Amazon S3 to stop multipart uploads that aren't completed within a specified number of days after being initiated. When a multipart upload isn't completed within the specified time frame, it becomes eligible for an abort operation. Amazon S3 then stops the multipart upload and deletes the parts associated with the multipart upload. This rule applies to both existing multipart uploads and those that you create later. The following is an example lifecycle configuration that specifies a rule with the `AbortIncompleteMultipartUpload` action. ``` sample-rule Enabled 7 ``` In the example, the rule doesn't specify a value for the `Prefix` element (the [object key name prefix](https://docs.aws.amazon.com/general/latest/gr/glos-chap.html#keyprefix)). Therefore, the rule applies to all objects in the bucket for which you initiated multipart uploads. Any multipart uploads that were initiated and weren't completed within seven days become eligible for an abort operation. The abort action has no effect on completed multipart uploads. For more information about the bucket lifecycle configuration, see [Managing the lifecycle of objects](object-lifecycle-mgmt.md). **Note** If the multipart upload is completed within the number of days specified in the rule, the `AbortIncompleteMultipartUpload` lifecycle action does not apply (that is, Amazon S3 doesn't take any action). Also, this action doesn't apply to objects. No objects are deleted by this lifecycle action. Additionally, you will not incur early delete charges for S3 Lifecycle when you remove any incomplete multipart upload parts. ## Using the S3 console To automatically manage incomplete multipart uploads, you can use the S3 console to create a lifecycle rule to expire incomplete multipart upload bytes from your bucket after a specified number of days. The following procedure shows you how to add a lifecycle rule to delete incomplete multipart uploads after 7 days. For more information about adding lifecycle rules, see [Setting an S3 Lifecycle configuration on a bucket](how-to-set-lifecycle-configuration-intro.md). **To add a lifecycle rule to abort incomplete multipart uploads that are more than 7 days old** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the **Buckets **list, choose the name of the bucket that you want to create a lifecycle rule for. 1. Choose the **Management** tab, and choose **Create lifecycle rule**. 1. In **Lifecycle rule name**, enter a name for your rule. The name must be unique within the bucket. 1. Choose the scope of the lifecycle rule: + To create a lifecycle rule for all objects with a specific prefix, choose **Limit the scope of this rule using one or more filters**, and enter the prefix in the **Prefix **field. + To create a lifecycle rule for all objects in the bucket, choose **This rule applies to **all** objects in the bucket**, and choose **I acknowledge that this rule applies to all objects in the bucket**. 1. Under **Lifecycle rule actions**, select **Delete expired object delete markers or incomplete multipart uploads**. 1. Under **Delete expired object delete markers or incomplete multipart uploads**, select **Delete incomplete multipart uploads**. 1. In the **Number of days **field, enter the number of days after which to delete incomplete multipart uploads (for this example, 7 days). 1. Choose **Create rule**. ## Using the AWS CLI The following `put-bucket-lifecycle-configuration` AWS Command Line Interface (AWS CLI) command adds the lifecycle configuration for the specified bucket. To use this command, replace the `user input placeholders` with your information. ``` aws s3api put-bucket-lifecycle-configuration  \ --bucket amzn-s3-demo-bucket  \ --lifecycle-configuration filename-containing-lifecycle-configuration ``` The following example shows how to add a lifecycle rule to abort incomplete multipart uploads by using the AWS CLI. It includes an example JSON lifecycle configuration to abort incomplete multipart uploads that are more than 7 days old. To use the CLI commands in this example, replace the `user input placeholders` with your information. **To add a lifecycle rule to abort incomplete multipart uploads** 1. Set up the AWS CLI. For instructions, see [Developing with Amazon S3 using the AWS CLI](https://docs.aws.amazon.com/AmazonS3/latest/API/setup-aws-cli.html) in the *Amazon S3 API Reference*. 1. Save the following example lifecycle configuration in a file (for example, *`lifecycle.json`*``). This example configuration specifies an empty prefix, and therefore it applies to all objects in the bucket. To restrict the configuration to a subset of objects, you can specify a prefix. ``` { "Rules": [ { "ID": "Test Rule", "Status": "Enabled", "Filter": { "Prefix": "" }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } } ] } ``` 1. Run the following CLI command to set this lifecycle configuration on your bucket. ``` aws s3api put-bucket-lifecycle-configuration   \ --bucket amzn-s3-demo-bucket  \ --lifecycle-configuration file://lifecycle.json ``` 1. To verify that the lifecycle configuration has been set on your bucket, retrieve the lifecycle configuration by using the following `get-bucket-lifecycle` command. ``` aws s3api get-bucket-lifecycle  \ --bucket amzn-s3-demo-bucket ``` 1. To delete the lifecycle configuration, use the following `delete-bucket-lifecycle` command. ``` aws s3api delete-bucket-lifecycle \ --bucket amzn-s3-demo-bucket ``` # Uploading an object using multipart upload You can use the multipart upload to programmatically upload a single object to Amazon S3. Each object is uploaded as a set of parts. Each part is a contiguous portion of the object's data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. Anonymous users cannot initiate multipart uploads. For an end-to-end procedure on uploading an object with multipart upload with an additional checksum, see [Tutorial: Upload an object through multipart upload and verify its data integrity](tutorial-s3-mpu-additional-checksums.md). The following section show how to use multipart upload with the AWS Command Line Interface, and AWS SDKs. ## Using the S3 console You can upload any file type—images, backups, data, movies, and so on—into an S3 bucket. The maximum size of a file that you can upload by using the Amazon S3 console is 160 GB. To upload a file larger than 160 GB, use the AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. For instructions on uploading an object via the AWS Management Console, see [Uploading objects](upload-objects.md). ## Using the AWS CLI The following describe the Amazon S3 operations for multipart upload using the AWS CLI. + [Initiate Multipart Upload](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/create-multipart-upload.html) + [Upload Part](https://docs.aws.amazon.com/cli/latest/reference/s3api/upload-part.html) + [Upload Part (Copy)](https://docs.aws.amazon.com/cli/latest/reference/s3api/upload-part-copy.html) + [Complete Multipart Upload](https://docs.aws.amazon.com/cli/latest/reference/s3api/complete-multipart-upload.html) + [Abort Multipart Upload](https://docs.aws.amazon.com/cli/latest/reference/s3api/abort-multipart-upload.html) + [List Parts](https://docs.aws.amazon.com/cli/latest/reference/s3api/list-parts.html) + [List Multipart Uploads](https://docs.aws.amazon.com/cli/latest/reference/s3api/list-multipart-uploads.html) ## Using the REST API The following sections in the *Amazon Simple Storage Service API Reference* describe the REST API for multipart upload. + [Initiate Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html) + [Upload Part](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html) + [Complete Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html) + [Stop Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadAbort.html) + [List Parts](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html) + [List Multipart Uploads](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html) ## Using the AWS SDKs (high-level API) Some AWS SDKs expose a high-level API that simplifies multipart upload by combining the different API operations required to complete a multipart upload into a single operation. For more information, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). If you need to pause and resume multipart uploads, vary part sizes during the upload, or do not know the size of the data in advance, use the low-level API methods. The low-level API methods for multipart uploads offer additional functionality, for more information, see [Using the AWS SDKs (low-level API)](#mpu-upload-low-level). ------ #### [ Java ] For examples of how to perform a multipart upload with the AWS SDK for Java, see [Upload or download large files to and from Amazon S3 using an AWS SDK](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_Scenario_UsingLargeFiles_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] To upload a file to an S3 bucket, use the `TransferUtility` class. When uploading data from a file, you must provide the object's key name. If you don't, the API uses the file name for the key name. When uploading data from a stream, you must provide the object's key name. To set advanced upload options—such as the part size, the number of threads when uploading the parts concurrently, metadata, the storage class, or ACL—use the `TransferUtilityUploadRequest` class. **Note** When you're using a stream for the source of data, the `TransferUtility` class does not do concurrent uploads. The following C\$1 example uploads a file to an Amazon S3 bucket in multiple parts. It shows how to use various `TransferUtility.Upload` overloads to upload a file. Each successive call to upload replaces the previous upload. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Transfer; using System; using System.IO; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class UploadFileMPUHighLevelAPITest { private const string bucketName = "*** provide bucket name ***"; private const string keyName = "*** provide a name for the uploaded object ***"; private const string filePath = "*** provide the full path name of the file to upload ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; public static void Main() { s3Client = new AmazonS3Client(bucketRegion); UploadFileAsync().Wait(); } private static async Task UploadFileAsync() { try { var fileTransferUtility = new TransferUtility(s3Client); // Option 1. Upload a file. The file name is used as the object key name. await fileTransferUtility.UploadAsync(filePath, bucketName); Console.WriteLine("Upload 1 completed"); // Option 2. Specify object key name explicitly. await fileTransferUtility.UploadAsync(filePath, bucketName, keyName); Console.WriteLine("Upload 2 completed"); // Option 3. Upload data from a type of System.IO.Stream. using (var fileToUpload = new FileStream(filePath, FileMode.Open, FileAccess.Read)) { await fileTransferUtility.UploadAsync(fileToUpload, bucketName, keyName); } Console.WriteLine("Upload 3 completed"); // Option 4. Specify advanced settings. var fileTransferUtilityRequest = new TransferUtilityUploadRequest { BucketName = bucketName, FilePath = filePath, StorageClass = S3StorageClass.StandardInfrequentAccess, PartSize = 6291456, // 6 MB. Key = keyName, CannedACL = S3CannedACL.PublicRead }; fileTransferUtilityRequest.Metadata.Add("param1", "Value1"); fileTransferUtilityRequest.Metadata.Add("param2", "Value2"); await fileTransferUtility.UploadAsync(fileTransferUtilityRequest); Console.WriteLine("Upload 4 completed"); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } } } } ``` ------ #### [ JavaScript ] **Example** Upload a large file. ``` import { S3Client } from "@aws-sdk/client-s3"; import { Upload } from "@aws-sdk/lib-storage"; import { ProgressBar, logger, } from "@aws-doc-sdk-examples/lib/utils/util-log.js"; const twentyFiveMB = 25 * 1024 * 1024; export const createString = (size = twentyFiveMB) => { return "x".repeat(size); }; /** * Create a 25MB file and upload it in parts to the specified * Amazon S3 bucket. * @param {{ bucketName: string, key: string }} */ export const main = async ({ bucketName, key }) => { const str = createString(); const buffer = Buffer.from(str, "utf8"); const progressBar = new ProgressBar({ description: `Uploading "${key}" to "${bucketName}"`, barLength: 30, }); try { const upload = new Upload({ client: new S3Client({}), params: { Bucket: bucketName, Key: key, Body: buffer, }, }); upload.on("httpUploadProgress", ({ loaded, total }) => { progressBar.update({ current: loaded, total }); }); await upload.done(); } catch (caught) { if (caught instanceof Error && caught.name === "AbortError") { logger.error(`Multipart upload was aborted. ${caught.message}`); } else { throw caught; } } }; ``` **Example** Download a large file. ``` import { fileURLToPath } from "node:url"; import { GetObjectCommand, NoSuchKey, S3Client } from "@aws-sdk/client-s3"; import { createWriteStream, rmSync } from "node:fs"; const s3Client = new S3Client({}); const oneMB = 1024 * 1024; export const getObjectRange = ({ bucket, key, start, end }) => { const command = new GetObjectCommand({ Bucket: bucket, Key: key, Range: `bytes=${start}-${end}`, }); return s3Client.send(command); }; /** * @param {string | undefined} contentRange */ export const getRangeAndLength = (contentRange) => { const [range, length] = contentRange.split("/"); const [start, end] = range.split("-"); return { start: Number.parseInt(start), end: Number.parseInt(end), length: Number.parseInt(length), }; }; export const isComplete = ({ end, length }) => end === length - 1; const downloadInChunks = async ({ bucket, key }) => { const writeStream = createWriteStream( fileURLToPath(new URL(`./${key}`, import.meta.url)), ).on("error", (err) => console.error(err)); let rangeAndLength = { start: -1, end: -1, length: -1 }; while (!isComplete(rangeAndLength)) { const { end } = rangeAndLength; const nextRange = { start: end + 1, end: end + oneMB }; const { ContentRange, Body } = await getObjectRange({ bucket, key, ...nextRange, }); console.log(`Downloaded bytes ${nextRange.start} to ${nextRange.end}`); writeStream.write(await Body.transformToByteArray()); rangeAndLength = getRangeAndLength(ContentRange); } }; /** * Download a large object from and Amazon S3 bucket. * * When downloading a large file, you might want to break it down into * smaller pieces. Amazon S3 accepts a Range header to specify the start * and end of the byte range to be downloaded. * * @param {{ bucketName: string, key: string }} */ export const main = async ({ bucketName, key }) => { try { await downloadInChunks({ bucket: bucketName, key: key, }); } catch (caught) { if (caught instanceof NoSuchKey) { console.error(`Failed to download object. No such key "${key}".`); rmSync(key); } } }; ``` ------ #### [ Go ] For more information about the Go code example for multipart upload, see [Upload or download large files to and from Amazon S3 using an AWS SDK](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_Scenario_UsingLargeFiles_section.html). **Example** Upload a large object by using an upload manager to break the data into parts and upload them concurrently. ``` import ( "bytes" "context" "errors" "fmt" "io" "log" "os" "time" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/feature/s3/manager" "github.com/aws/aws-sdk-go-v2/service/s3" "github.com/aws/aws-sdk-go-v2/service/s3/types" "github.com/aws/smithy-go" ) // BucketBasics encapsulates the Amazon Simple Storage Service (Amazon S3) actions // used in the examples. // It contains S3Client, an Amazon S3 service client that is used to perform bucket // and object actions. type BucketBasics struct { S3Client *s3.Client } ``` ``` // UploadLargeObject uses an upload manager to upload data to an object in a bucket. // The upload manager breaks large data into parts and uploads the parts concurrently. func (basics BucketBasics) UploadLargeObject(ctx context.Context, bucketName string, objectKey string, largeObject []byte) error { largeBuffer := bytes.NewReader(largeObject) var partMiBs int64 = 10 uploader := manager.NewUploader(basics.S3Client, func(u *manager.Uploader) { u.PartSize = partMiBs * 1024 * 1024 }) _, err := uploader.Upload(ctx, &s3.PutObjectInput{ Bucket: aws.String(bucketName), Key: aws.String(objectKey), Body: largeBuffer, }) if err != nil { var apiErr smithy.APIError if errors.As(err, &apiErr) && apiErr.ErrorCode() == "EntityTooLarge" { log.Printf("Error while uploading object to %s. The object is too large.\n"+ "The maximum size for a multipart upload is 5TB.", bucketName) } else { log.Printf("Couldn't upload large object to %v:%v. Here's why: %v\n", bucketName, objectKey, err) } } else { err = s3.NewObjectExistsWaiter(basics.S3Client).Wait( ctx, &s3.HeadObjectInput{Bucket: aws.String(bucketName), Key: aws.String(objectKey)}, time.Minute) if err != nil { log.Printf("Failed attempt to wait for object %s to exist.\n", objectKey) } } return err } ``` **Example** Download a large object by using a download manager to get the data in parts and download them concurrently. ``` // DownloadLargeObject uses a download manager to download an object from a bucket. // The download manager gets the data in parts and writes them to a buffer until all of // the data has been downloaded. func (basics BucketBasics) DownloadLargeObject(ctx context.Context, bucketName string, objectKey string) ([]byte, error) { var partMiBs int64 = 10 downloader := manager.NewDownloader(basics.S3Client, func(d *manager.Downloader) { d.PartSize = partMiBs * 1024 * 1024 }) buffer := manager.NewWriteAtBuffer([]byte{}) _, err := downloader.Download(ctx, buffer, &s3.GetObjectInput{ Bucket: aws.String(bucketName), Key: aws.String(objectKey), }) if err != nil { log.Printf("Couldn't download large object from %v:%v. Here's why: %v\n", bucketName, objectKey, err) } return buffer.Bytes(), err } ``` ------ #### [ PHP ] This topic explains how to use the high-level `Aws\S3\Model\MultipartUpload\UploadBuilder` class from the AWS SDK for PHP for multipart file uploads. For more information about the AWS SDK for Ruby API, go to [AWS SDK for Ruby - Version 2](https://docs.aws.amazon.com/sdkforruby/api/index.html). The following PHP example uploads a file to an Amazon S3 bucket. The example demonstrates how to set parameters for the `MultipartUploader` object. ``` require 'vendor/autoload.php'; use Aws\Exception\MultipartUploadException; use Aws\S3\MultipartUploader; use Aws\S3\S3Client; $bucket = '*** Your Bucket Name ***'; $keyname = '*** Your Object Key ***'; $s3 = new S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); // Prepare the upload parameters. $uploader = new MultipartUploader($s3, '/path/to/large/file.zip', [ 'bucket' => $bucket, 'key' => $keyname ]); // Perform the upload. try { $result = $uploader->upload(); echo "Upload complete: {$result['ObjectURL']}" . PHP_EOL; } catch (MultipartUploadException $e) { echo $e->getMessage() . PHP_EOL; } ``` ------ #### [ Python ] The following example loads an object using the high-level multipart upload Python API (the `TransferManager` class). ``` import sys import threading import boto3 from boto3.s3.transfer import TransferConfig MB = 1024 * 1024 s3 = boto3.resource("s3") class TransferCallback: """ Handle callbacks from the transfer manager. The transfer manager periodically calls the __call__ method throughout the upload and download process so that it can take action, such as displaying progress to the user and collecting data about the transfer. """ def __init__(self, target_size): self._target_size = target_size self._total_transferred = 0 self._lock = threading.Lock() self.thread_info = {} def __call__(self, bytes_transferred): """ The callback method that is called by the transfer manager. Display progress during file transfer and collect per-thread transfer data. This method can be called by multiple threads, so shared instance data is protected by a thread lock. """ thread = threading.current_thread() with self._lock: self._total_transferred += bytes_transferred if thread.ident not in self.thread_info.keys(): self.thread_info[thread.ident] = bytes_transferred else: self.thread_info[thread.ident] += bytes_transferred target = self._target_size * MB sys.stdout.write( f"\r{self._total_transferred} of {target} transferred " f"({(self._total_transferred / target) * 100:.2f}%)." ) sys.stdout.flush() def upload_with_default_configuration( local_file_path, bucket_name, object_key, file_size_mb ): """ Upload a file from a local folder to an Amazon S3 bucket, using the default configuration. """ transfer_callback = TransferCallback(file_size_mb) s3.Bucket(bucket_name).upload_file( local_file_path, object_key, Callback=transfer_callback ) return transfer_callback.thread_info def upload_with_chunksize_and_meta( local_file_path, bucket_name, object_key, file_size_mb, metadata=None ): """ Upload a file from a local folder to an Amazon S3 bucket, setting a multipart chunk size and adding metadata to the Amazon S3 object. The multipart chunk size controls the size of the chunks of data that are sent in the request. A smaller chunk size typically results in the transfer manager using more threads for the upload. The metadata is a set of key-value pairs that are stored with the object in Amazon S3. """ transfer_callback = TransferCallback(file_size_mb) config = TransferConfig(multipart_chunksize=1 * MB) extra_args = {"Metadata": metadata} if metadata else None s3.Bucket(bucket_name).upload_file( local_file_path, object_key, Config=config, ExtraArgs=extra_args, Callback=transfer_callback, ) return transfer_callback.thread_info def upload_with_high_threshold(local_file_path, bucket_name, object_key, file_size_mb): """ Upload a file from a local folder to an Amazon S3 bucket, setting a multipart threshold larger than the size of the file. Setting a multipart threshold larger than the size of the file results in the transfer manager sending the file as a standard upload instead of a multipart upload. """ transfer_callback = TransferCallback(file_size_mb) config = TransferConfig(multipart_threshold=file_size_mb * 2 * MB) s3.Bucket(bucket_name).upload_file( local_file_path, object_key, Config=config, Callback=transfer_callback ) return transfer_callback.thread_info def upload_with_sse( local_file_path, bucket_name, object_key, file_size_mb, sse_key=None ): """ Upload a file from a local folder to an Amazon S3 bucket, adding server-side encryption with customer-provided encryption keys to the object. When this kind of encryption is specified, Amazon S3 encrypts the object at rest and allows downloads only when the expected encryption key is provided in the download request. """ transfer_callback = TransferCallback(file_size_mb) if sse_key: extra_args = {"SSECustomerAlgorithm": "AES256", "SSECustomerKey": sse_key} else: extra_args = None s3.Bucket(bucket_name).upload_file( local_file_path, object_key, ExtraArgs=extra_args, Callback=transfer_callback ) return transfer_callback.thread_info def download_with_default_configuration( bucket_name, object_key, download_file_path, file_size_mb ): """ Download a file from an Amazon S3 bucket to a local folder, using the default configuration. """ transfer_callback = TransferCallback(file_size_mb) s3.Bucket(bucket_name).Object(object_key).download_file( download_file_path, Callback=transfer_callback ) return transfer_callback.thread_info def download_with_single_thread( bucket_name, object_key, download_file_path, file_size_mb ): """ Download a file from an Amazon S3 bucket to a local folder, using a single thread. """ transfer_callback = TransferCallback(file_size_mb) config = TransferConfig(use_threads=False) s3.Bucket(bucket_name).Object(object_key).download_file( download_file_path, Config=config, Callback=transfer_callback ) return transfer_callback.thread_info def download_with_high_threshold( bucket_name, object_key, download_file_path, file_size_mb ): """ Download a file from an Amazon S3 bucket to a local folder, setting a multipart threshold larger than the size of the file. Setting a multipart threshold larger than the size of the file results in the transfer manager sending the file as a standard download instead of a multipart download. """ transfer_callback = TransferCallback(file_size_mb) config = TransferConfig(multipart_threshold=file_size_mb * 2 * MB) s3.Bucket(bucket_name).Object(object_key).download_file( download_file_path, Config=config, Callback=transfer_callback ) return transfer_callback.thread_info def download_with_sse( bucket_name, object_key, download_file_path, file_size_mb, sse_key ): """ Download a file from an Amazon S3 bucket to a local folder, adding a customer-provided encryption key to the request. When this kind of encryption is specified, Amazon S3 encrypts the object at rest and allows downloads only when the expected encryption key is provided in the download request. """ transfer_callback = TransferCallback(file_size_mb) if sse_key: extra_args = {"SSECustomerAlgorithm": "AES256", "SSECustomerKey": sse_key} else: extra_args = None s3.Bucket(bucket_name).Object(object_key).download_file( download_file_path, ExtraArgs=extra_args, Callback=transfer_callback ) return transfer_callback.thread_info ``` ------ ## Using the AWS SDKs (low-level API) The AWS SDK exposes a low-level API that closely resembles the Amazon S3 REST API for multipart uploads (see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). Use the low-level API when you need to pause and resume multipart uploads, vary part sizes during the upload, or do not know the size of the upload data in advance. When you don't have these requirements, use the high-level API (see [Using the AWS SDKs (high-level API)](#multipart-upload-high-level)). ------ #### [ Java ] The following example shows how to use the low-level Java classes to upload a file. It performs the following steps: + Initiates a multipart upload using the `AmazonS3Client.initiateMultipartUpload()` method, and passes in an `InitiateMultipartUploadRequest` object. + Saves the upload ID that the `AmazonS3Client.initiateMultipartUpload()` method returns. You provide this upload ID for each subsequent multipart upload operation. + Uploads the parts of the object. For each part, you call the `AmazonS3Client.uploadPart()` method. You provide part upload information using an `UploadPartRequest` object. + For each part, saves the ETag from the response of the `AmazonS3Client.uploadPart()` method in a list. You use the ETag values to complete the multipart upload. + Calls the `AmazonS3Client.completeMultipartUpload()` method to complete the multipart upload. **Example** For instructions on creating and testing a working sample, see [Getting Started](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/getting-started.html) in the AWS SDK for Java Developer Guide. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.auth.profile.ProfileCredentialsProvider; import com.amazonaws.regions.Regions; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.*; import java.io.File; import java.io.IOException; import java.util.ArrayList; import java.util.List; public class LowLevelMultipartUpload { public static void main(String[] args) throws IOException { Regions clientRegion = Regions.DEFAULT_REGION; String bucketName = "*** Bucket name ***"; String keyName = "*** Key name ***"; String filePath = "*** Path to file to upload ***"; File file = new File(filePath); long contentLength = file.length(); long partSize = 5 * 1024 * 1024; // Set part size to 5 MB. try { AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .withRegion(clientRegion) .withCredentials(new ProfileCredentialsProvider()) .build(); // Create a list of ETag objects. You retrieve ETags for each object part // uploaded, // then, after each individual part has been uploaded, pass the list of ETags to // the request to complete the upload. List partETags = new ArrayList(); // Initiate the multipart upload. InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, keyName); InitiateMultipartUploadResult initResponse = s3Client.initiateMultipartUpload(initRequest); // Upload the file parts. long filePosition = 0; for (int i = 1; filePosition < contentLength; i++) { // Because the last part could be less than 5 MB, adjust the part size as // needed. partSize = Math.min(partSize, (contentLength - filePosition)); // Create the request to upload a part. UploadPartRequest uploadRequest = new UploadPartRequest() .withBucketName(bucketName) .withKey(keyName) .withUploadId(initResponse.getUploadId()) .withPartNumber(i) .withFileOffset(filePosition) .withFile(file) .withPartSize(partSize); // Upload the part and add the response's ETag to our list. UploadPartResult uploadResult = s3Client.uploadPart(uploadRequest); partETags.add(uploadResult.getPartETag()); filePosition += partSize; } // Complete the multipart upload. CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, keyName, initResponse.getUploadId(), partETags); s3Client.completeMultipartUpload(compRequest); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` ------ #### [ .NET ] The following C\$1 example shows how to use the low-level SDK for .NET multipart upload API to upload a file to an S3 bucket. For information about Amazon S3 multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). **Note** When you use the SDK for .NET API to upload large objects, a timeout might occur while data is being written to the request stream. You can set an explicit timeout using the `UploadPartRequest`. The following C\$1 example uploads a file to an S3 bucket using the low-level multipart upload API. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.Runtime; using Amazon.S3; using Amazon.S3.Model; using System; using System.Collections.Generic; using System.IO; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class UploadFileMPULowLevelAPITest { private const string bucketName = "*** provide bucket name ***"; private const string keyName = "*** provide a name for the uploaded object ***"; private const string filePath = "*** provide the full path name of the file to upload ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; public static void Main() { s3Client = new AmazonS3Client(bucketRegion); Console.WriteLine("Uploading an object"); UploadObjectAsync().Wait(); } private static async Task UploadObjectAsync() { // Create list to store upload part responses. List uploadResponses = new List(); // Setup information required to initiate the multipart upload. InitiateMultipartUploadRequest initiateRequest = new InitiateMultipartUploadRequest { BucketName = bucketName, Key = keyName }; // Initiate the upload. InitiateMultipartUploadResponse initResponse = await s3Client.InitiateMultipartUploadAsync(initiateRequest); // Upload parts. long contentLength = new FileInfo(filePath).Length; long partSize = 5 * (long)Math.Pow(2, 20); // 5 MB try { Console.WriteLine("Uploading parts"); long filePosition = 0; for (int i = 1; filePosition < contentLength; i++) { UploadPartRequest uploadRequest = new UploadPartRequest { BucketName = bucketName, Key = keyName, UploadId = initResponse.UploadId, PartNumber = i, PartSize = partSize, FilePosition = filePosition, FilePath = filePath }; // Track upload progress. uploadRequest.StreamTransferProgress += new EventHandler(UploadPartProgressEventCallback); // Upload a part and add the response to our list. uploadResponses.Add(await s3Client.UploadPartAsync(uploadRequest)); filePosition += partSize; } // Setup to complete the upload. CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest { BucketName = bucketName, Key = keyName, UploadId = initResponse.UploadId }; completeRequest.AddPartETags(uploadResponses); // Complete the upload. CompleteMultipartUploadResponse completeUploadResponse = await s3Client.CompleteMultipartUploadAsync(completeRequest); } catch (Exception exception) { Console.WriteLine("An AmazonS3Exception was thrown: { 0}", exception.Message); // Abort the upload. AbortMultipartUploadRequest abortMPURequest = new AbortMultipartUploadRequest { BucketName = bucketName, Key = keyName, UploadId = initResponse.UploadId }; await s3Client.AbortMultipartUploadAsync(abortMPURequest); } } public static void UploadPartProgressEventCallback(object sender, StreamTransferProgressArgs e) { // Process event. Console.WriteLine("{0}/{1}", e.TransferredBytes, e.TotalBytes); } } } ``` ------ #### [ PHP ] This topic shows how to use the low-level `uploadPart` method from version 3 of the AWS SDK for PHP to upload a file in multiple parts. For more information about the AWS SDK for Ruby API, go to [AWS SDK for Ruby - Version 2](https://docs.aws.amazon.com/sdkforruby/api/index.html). The following PHP example uploads a file to an Amazon S3 bucket using the low-level PHP API multipart upload. ``` require 'vendor/autoload.php'; use Aws\S3\Exception\S3Exception; use Aws\S3\S3Client; $bucket = '*** Your Bucket Name ***'; $keyname = '*** Your Object Key ***'; $filename = '*** Path to and Name of the File to Upload ***'; $s3 = new S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); $result = $s3->createMultipartUpload([ 'Bucket' => $bucket, 'Key' => $keyname, 'StorageClass' => 'REDUCED_REDUNDANCY', 'Metadata' => [ 'param1' => 'value 1', 'param2' => 'value 2', 'param3' => 'value 3' ] ]); $uploadId = $result['UploadId']; // Upload the file in parts. try { $file = fopen($filename, 'r'); $partNumber = 1; while (!feof($file)) { $result = $s3->uploadPart([ 'Bucket' => $bucket, 'Key' => $keyname, 'UploadId' => $uploadId, 'PartNumber' => $partNumber, 'Body' => fread($file, 5 * 1024 * 1024), ]); $parts['Parts'][$partNumber] = [ 'PartNumber' => $partNumber, 'ETag' => $result['ETag'], ]; $partNumber++; echo "Uploading part $partNumber of $filename." . PHP_EOL; } fclose($file); } catch (S3Exception $e) { $result = $s3->abortMultipartUpload([ 'Bucket' => $bucket, 'Key' => $keyname, 'UploadId' => $uploadId ]); echo "Upload of $filename failed." . PHP_EOL; } // Complete the multipart upload. $result = $s3->completeMultipartUpload([ 'Bucket' => $bucket, 'Key' => $keyname, 'UploadId' => $uploadId, 'MultipartUpload' => $parts, ]); $url = $result['Location']; echo "Uploaded $filename to $url." . PHP_EOL; ``` ------ ## Using the AWS SDK for Ruby The AWS SDK for Ruby version 3 supports Amazon S3 multipart uploads in two ways. For the first option, you can use managed file uploads. For more information, see [Uploading Files to Amazon S3](https://aws.amazon.com/blogs/developer/uploading-files-to-amazon-s3/) in the *AWS Developer Blog*. Managed file uploads are the recommended method for uploading files to a bucket. They provide the following benefits: + Manage multipart uploads for objects larger than 15MB. + Correctly open files in binary mode to avoid encoding issues. + Use multiple threads for uploading parts of large objects in parallel. Alternatively, you can use the following multipart upload client operations directly: + [create\$1multipart\$1upload](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#create_multipart_upload-instance_method) – Initiates a multipart upload and returns an upload ID. + [upload\$1part](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#upload_part-instance_method) – Uploads a part in a multipart upload. + [upload\$1part\$1copy](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#upload_part_copy-instance_method) – Uploads a part by copying data from an existing object as data source. + [complete\$1multipart\$1upload](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#complete_multipart_upload-instance_method) – Completes a multipart upload by assembling previously uploaded parts. + [abort\$1multipart\$1upload](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#abort_multipart_upload-instance_method) – Stops a multipart upload. # Uploading a directory using the high-level .NET TransferUtility class You can use the `TransferUtility` class to upload an entire directory. By default, the API uploads only the files at the root of the specified directory. You can, however, specify recursively uploading files in all of the sub directories. To select files in the specified directory based on filtering criteria, specify filtering expressions. For example, to upload only the `PDF` files from a directory, specify the `"*.pdf"` filter expression. When uploading files from a directory, you don't specify the key names for the resulting objects. Amazon S3 constructs the key names using the original file path. For example, assume that you have a directory called `c:\myfolder` with the following structure: **Example** ``` 1. C:\myfolder 2. \a.txt 3. \b.pdf 4. \media\ 5. An.mp3 ``` When you upload this directory, Amazon S3 uses the following key names: **Example** ``` 1. a.txt 2. b.pdf 3. media/An.mp3 ``` **Example** The following C\$1 example uploads a directory to an Amazon S3 bucket. It shows how to use various `TransferUtility.UploadDirectory` overloads to upload the directory. Each successive call to upload replaces the previous upload. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Transfer; using System; using System.IO; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class UploadDirMPUHighLevelAPITest { private const string existingBucketName = "*** bucket name ***"; private const string directoryPath = @"*** directory path ***"; // The example uploads only .txt files. private const string wildCard = "*.txt"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; static void Main() { s3Client = new AmazonS3Client(bucketRegion); UploadDirAsync().Wait(); } private static async Task UploadDirAsync() { try { var directoryTransferUtility = new TransferUtility(s3Client); // 1. Upload a directory. await directoryTransferUtility.UploadDirectoryAsync(directoryPath, existingBucketName); Console.WriteLine("Upload statement 1 completed"); // 2. Upload only the .txt files from a directory // and search recursively. await directoryTransferUtility.UploadDirectoryAsync( directoryPath, existingBucketName, wildCard, SearchOption.AllDirectories); Console.WriteLine("Upload statement 2 completed"); // 3. The same as Step 2 and some optional configuration. // Search recursively for .txt files to upload. var request = new TransferUtilityUploadDirectoryRequest { BucketName = existingBucketName, Directory = directoryPath, SearchOption = SearchOption.AllDirectories, SearchPattern = wildCard }; await directoryTransferUtility.UploadDirectoryAsync(request); Console.WriteLine("Upload statement 3 completed"); } catch (AmazonS3Exception e) { Console.WriteLine( "Error encountered ***. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine( "Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } } } } ``` # Listing multipart uploads You can use the AWS CLI, REST API, or AWS SDKs, to retrieve a list of in-progress multipart uploads in Amazon S3. You can use the multipart upload to programmatically upload a single object to Amazon S3. Multipart uploads move objects into Amazon S3 by moving a portion of an object's data at a time. For more general information about multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). For an end-to-end procedure on uploading an object with multipart upload with an additional checksum, see [Tutorial: Upload an object through multipart upload and verify its data integrity](tutorial-s3-mpu-additional-checksums.md). The following section show how to list in-progress multipart uploads with the AWS Command Line Interface, the Amazon S3 REST API, and AWS SDKs. ## Listing multipart uploads using the AWS CLI The following sections in the AWS Command Line Interface describe the operations for listing multipart uploads. + [list-parts](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/list-parts.html)‐list the uploaded parts for a specific multipart upload. + [list-multipart-uploads](https://docs.aws.amazon.com/cli/latest/reference/s3api/list-multipart-uploads.html)‐list in-progress multipart uploads. # Listing multipart uploads using the REST API The following sections in the *Amazon Simple Storage Service API Reference* describe the REST API for listing multipart uploads: + [ListParts](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html)‐list the uploaded parts for a specific multipart upload. + [ListMultipartUploads](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html)‐list in-progress multipart uploads. ## Listing multipart uploads using the AWS SDK (low-level API) ------ #### [ Java ] To list all in-progress multipart uploads on a bucket using the AWS SDK for Java, you can use the low-level API classes to: **Low-level API multipart uploads listing process** | | | | --- |--- | | 1 | Create an instance of the `ListMultipartUploadsRequest` class and provide the bucket name. | | 2 | Run the S3Client `listMultipartUploads` method. The method returns an instance of the `ListMultipartUploadsResponse` class that gives you information about the multipart uploads in progress. | For examples of how to list multipart uploads with the AWS SDK for Java, see [List multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_ListMultipartUploads_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] To list all of the in-progress multipart uploads on a specific bucket, use the SDK for .NET low-level multipart upload API's `ListMultipartUploadsRequest` class. The `AmazonS3Client.ListMultipartUploads` method returns an instance of the `ListMultipartUploadsResponse` class that provides information about the in-progress multipart uploads. An in-progress multipart upload is a multipart upload that has been initiated using the initiate multipart upload request, but has not yet been completed or stopped. For more information about Amazon S3 multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). The following C\$1 example shows how to use the SDK for .NET to list all in-progress multipart uploads on a bucket. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` ListMultipartUploadsRequest request = new ListMultipartUploadsRequest { BucketName = bucketName // Bucket receiving the uploads. }; ListMultipartUploadsResponse response = await AmazonS3Client.ListMultipartUploadsAsync(request); ``` ------ #### [ PHP ] This topic shows how to use the low-level API classes from version 3 of the AWS SDK for PHP to list all in-progress multipart uploads on a bucket. For more information about the AWS SDK for Ruby API, go to [AWS SDK for Ruby - Version 2](https://docs.aws.amazon.com/sdkforruby/api/index.html). The following PHP example demonstrates listing all in-progress multipart uploads on a bucket. ``` require 'vendor/autoload.php'; use Aws\S3\S3Client; $bucket = '*** Your Bucket Name ***'; $s3 = new S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); // Retrieve a list of the current multipart uploads. $result = $s3->listMultipartUploads([ 'Bucket' => $bucket ]); // Write the list of uploads to the page. print_r($result->toArray()); ``` ------ # Tracking a multipart upload with the AWS SDKs You can track an object's upload progress to Amazon S3 with a listen interface. The high-level multipart upload API provides such a listen interface, called `ProgressListener`. Progress events occur periodically and notify the listener that bytes have been transferred. For more general information about multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). For an end-to-end procedure on uploading an object with multipart upload with an additional checksum, see [Tutorial: Upload an object through multipart upload and verify its data integrity](tutorial-s3-mpu-additional-checksums.md). The following section show how to track a multipart upload with the AWS SDKs. ------ #### [ Java ] **Example** The following Java code uploads a file and uses the `ExecutionInterceptor` to track the upload progress. For instructions on how to create and test a working sample, see [Getting Started](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html) in the AWS SDK for Java 2.x Developer Guide. ``` import java.nio.file.Paths; import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider; import software.amazon.awssdk.core.async.AsyncRequestBody; import software.amazon.awssdk.core.interceptor.Context; import software.amazon.awssdk.core.interceptor.ExecutionAttributes; import software.amazon.awssdk.core.interceptor.ExecutionInterceptor; import software.amazon.awssdk.services.s3.S3AsyncClient; import software.amazon.awssdk.services.s3.model.PutObjectRequest; public class TrackMPUProgressUsingHighLevelAPI { static class ProgressListener implements ExecutionInterceptor { private long transferredBytes = 0; @Override public void beforeTransmission(Context.BeforeTransmission context, ExecutionAttributes executionAttributes) { if (context.httpRequest().firstMatchingHeader("Content-Length").isPresent()) { String contentLength = context.httpRequest().firstMatchingHeader("Content-Length").get(); long partSize = Long.parseLong(contentLength); transferredBytes += partSize; System.out.println("Transferred bytes: " + transferredBytes); } } } public static void main(String[] args) throws Exception { String existingBucketName = "*** Provide bucket name ***"; String keyName = "*** Provide object key ***"; String filePath = "*** file to upload ***"; S3AsyncClient s3Client = S3AsyncClient.builder() .credentialsProvider(ProfileCredentialsProvider.create()) .overrideConfiguration(c -> c.addExecutionInterceptor(new ProgressListener())) .build(); // For more advanced uploads, you can create a request object // and supply additional request parameters (ex: progress listeners, // canned ACLs, etc.) PutObjectRequest request = PutObjectRequest.builder() .bucket(existingBucketName) .key(keyName) .build(); AsyncRequestBody requestBody = AsyncRequestBody.fromFile(Paths.get(filePath)); // You can ask the upload for its progress, or you can // add a ProgressListener to your request to receive notifications // when bytes are transferred. // S3AsyncClient processes all transfers asynchronously, // so this call will return immediately. var upload = s3Client.putObject(request, requestBody); try { // You can block and wait for the upload to finish upload.join(); } catch (Exception exception) { System.out.println("Unable to upload file, upload aborted."); exception.printStackTrace(); } finally { s3Client.close(); } } } ``` ------ #### [ .NET ] The following C\$1 example uploads a file to an S3 bucket using the `TransferUtility` class, and tracks the progress of the upload. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Transfer; using System; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class TrackMPUUsingHighLevelAPITest { private const string bucketName = "*** provide the bucket name ***"; private const string keyName = "*** provide the name for the uploaded object ***"; private const string filePath = " *** provide the full path name of the file to upload **"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; public static void Main() { s3Client = new AmazonS3Client(bucketRegion); TrackMPUAsync().Wait(); } private static async Task TrackMPUAsync() { try { var fileTransferUtility = new TransferUtility(s3Client); // Use TransferUtilityUploadRequest to configure options. // In this example we subscribe to an event. var uploadRequest = new TransferUtilityUploadRequest { BucketName = bucketName, FilePath = filePath, Key = keyName }; uploadRequest.UploadProgressEvent += new EventHandler (uploadRequest_UploadPartProgressEvent); await fileTransferUtility.UploadAsync(uploadRequest); Console.WriteLine("Upload completed"); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } } static void uploadRequest_UploadPartProgressEvent(object sender, UploadProgressArgs e) { // Process event. Console.WriteLine("{0}/{1}", e.TransferredBytes, e.TotalBytes); } } } ``` ------ # Aborting a multipart upload After you initiate a multipart upload, you begin uploading parts. Amazon S3 stores these parts, and only creates the object after you upload all parts and send a request to complete the multipart upload. Upon receiving the complete multipart upload request, Amazon S3 assembles the parts and creates an object. If you don't send the complete multipart upload request successfully, S3 does not assemble the parts and does not create any object. If you wish to not complete a multipart upload after uploading parts you should abort the multipart upload. You are billed for all storage associated with uploaded parts. It's recommended to always either complete the multipart upload or stop the multipart upload to remove any uploaded parts. For more information about pricing, see [Multipart upload and pricing](mpuoverview.md#mpuploadpricing). You can also stop an incomplete multipart upload using a bucket lifecycle configuration. For more information, see [Configuring a bucket lifecycle configuration to delete incomplete multipart uploads](mpu-abort-incomplete-mpu-lifecycle-config.md). The following section show how to stop an in-progress multipart upload in Amazon S3 using the AWS Command Line Interface, REST API, or AWS SDKs. ## Using the AWS CLI For more information about using the AWS CLI to stop a multipart upload, see [abort-multipart-upload](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/abort-multipart-upload.html) in the *AWS CLI Command Reference*. ## Using the REST API For more information about using the REST API to stop a multipart upload, see [AbortMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) in the *Amazon Simple Storage Service API Reference*. ## Using the AWS SDKs (high-level API) ------ #### [ Java ] To stop multipart uploads in progress using the AWS SDK for Java, you can abort uploads that were initiated before a specified date and are still in progress. An upload is considered to be in progress after you initiate it and until you complete it or stop it. To stop multipart uploads, you can: | | | | --- |--- | | 1 | Create an S3Client instance. | | 2 | Use the client's abort methods by passing the bucket name and other required parameters. | **Note** You can also stop a specific multipart upload. For more information, see [Using the AWS SDKs (low-level API)](#abort-mpu-low-level). For examples of how to abort multipart uploads with the AWS SDK for Java, see [Cancel a multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_AbortMultipartUpload_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] The following C\$1 example stops all in-progress multipart uploads that were initiated on a specific bucket over a week ago. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Transfer; using System; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class AbortMPUUsingHighLevelAPITest { private const string bucketName = "*** provide bucket name ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; public static void Main() { s3Client = new AmazonS3Client(bucketRegion); AbortMPUAsync().Wait(); } private static async Task AbortMPUAsync() { try { var transferUtility = new TransferUtility(s3Client); // Abort all in-progress uploads initiated before the specified date. await transferUtility.AbortMultipartUploadsAsync( bucketName, DateTime.Now.AddDays(-7)); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } } } } ``` **Note** You can also stop a specific multipart upload. For more information, see [Using the AWS SDKs (low-level API)](#abort-mpu-low-level). ------ ## Using the AWS SDKs (low-level API) You can stop an in-progress multipart upload by calling the `AmazonS3.abortMultipartUpload` method. This method deletes any parts that were uploaded to Amazon S3 and frees up the resources. You must provide the upload ID, bucket name, and key name. The following Java code example demonstrates how to stop an in-progress multipart upload. To stop a multipart upload, you provide the upload ID, and the bucket and key names that are used in the upload. After you have stopped a multipart upload, you can't use the upload ID to upload additional parts. For more information about Amazon S3 multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). ------ #### [ Java ] To stop a specific in-progress multipart upload using the AWS SDK for Java, you can use the low-level API to abort the upload by providing the bucket name, object key, and upload ID. **Note** Instead of aborting a specific multipart upload, you can stop all multipart uploads initiated before a specific time that are still in progress. This clean-up operation is useful to stop old multipart uploads that you initiated but did not complete or stop. For more information, see [Using the AWS SDKs (high-level API)](#abort-mpu-high-level). For examples of how to abort a specific multipart upload with the AWS SDK for Java, see [Cancel a multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_AbortMultipartUpload_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] The following C\$1 example shows how to stop a multipart upload. For a complete C\$1 sample that includes the following code, see [Using the AWS SDKs (low-level API)](mpu-upload-object.md#mpu-upload-low-level). ``` AbortMultipartUploadRequest abortMPURequest = new AbortMultipartUploadRequest { BucketName = existingBucketName, Key = keyName, UploadId = initResponse.UploadId }; await AmazonS3Client.AbortMultipartUploadAsync(abortMPURequest); ``` You can also abort all in-progress multipart uploads that were initiated prior to a specific time. This clean-up operation is useful for aborting multipart uploads that didn't complete or were aborted. For more information, see [Using the AWS SDKs (high-level API)](#abort-mpu-high-level). ------ #### [ PHP ] This example shows how to use a class from version 3 of the AWS SDK for PHP to abort a multipart upload that is in progress. For more information about the AWS SDK for Ruby API, go to [AWS SDK for Ruby - Version 2](https://docs.aws.amazon.com/sdkforruby/api/index.html). The example the `abortMultipartUpload()` method. For more information about the AWS SDK for Ruby API, go to [AWS SDK for Ruby - Version 2](https://docs.aws.amazon.com/sdkforruby/api/index.html). ``` require 'vendor/autoload.php'; use Aws\S3\S3Client; $bucket = '*** Your Bucket Name ***'; $keyname = '*** Your Object Key ***'; $uploadId = '*** Upload ID of upload to Abort ***'; $s3 = new S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); // Abort the multipart upload. $s3->abortMultipartUpload([ 'Bucket' => $bucket, 'Key' => $keyname, 'UploadId' => $uploadId, ]); ``` ------ # Copying an object using multipart upload Multipart upload allows you to copy objects as a set of parts. The examples in this section show you how to copy objects greater than 5 GB using the multipart upload API. For information about multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). You can copy objects less than 5 GB in a single operation without using the multipart upload API. You can copy objects less than 5 GB using the AWS Management Console, AWS CLI, REST API, or AWS SDKs. For more information, see [Copying, moving, and renaming objects](copy-object.md). For an end-to-end procedure on uploading an object with multipart upload with an additional checksum, see [Tutorial: Upload an object through multipart upload and verify its data integrity](tutorial-s3-mpu-additional-checksums.md). The following section show how to copy an object with multipart upload with the REST API or AWS SDKs. ## Using the REST API The following sections in the *Amazon Simple Storage Service API Reference* describe the REST API for multipart upload. For copying an existing object, use the Upload Part (Copy) API and specify the source object by adding the `x-amz-copy-source` request header in your request. + [Initiate Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html) + [Upload Part](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html) + [Upload Part (Copy)](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html) + [Complete Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html) + [Abort Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadAbort.html) + [List Parts](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html) + [List Multipart Uploads](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html) You can use these APIs to make your own REST requests, or you can use one of the SDKs we provide. For more information about using Multipart Upload with the AWS CLI, see [Using the AWS CLI](mpu-upload-object.md#UsingCLImpUpload). For more information about the SDKs, see [AWS SDK support for multipart upload](mpuoverview.md#sdksupportformpu). ## Using the AWS SDKs To copy an object using the low-level API, do the following: + Initiate a multipart upload by calling the `AmazonS3Client.initiateMultipartUpload()` method. + Save the upload ID from the response object that the `AmazonS3Client.initiateMultipartUpload()` method returns. You provide this upload ID for each part-upload operation. + Copy all of the parts. For each part that you need to copy, create a new instance of the `CopyPartRequest` class. Provide the part information, including the source and destination bucket names, source and destination object keys, upload ID, locations of the first and last bytes of the part, and part number. + Save the responses of the `AmazonS3Client.copyPart()` method calls. Each response includes the `ETag` value and part number for the uploaded part. You need this information to complete the multipart upload. + Call the `AmazonS3Client.completeMultipartUpload()` method to complete the copy operation. ------ #### [ Java ] For examples of how to copy objects using multipart upload with the AWS SDK for Java, see [Copy part of an object from another object](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_UploadPartCopy_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] The following C\$1 example shows how to use the SDK for .NET to copy an Amazon S3 object that is larger than 5 GB from one source location to another, such as from one bucket to another. To copy objects that are smaller than 5 GB, use the single-operation copy procedure described in [Using the AWS SDKs](copy-object.md#CopyingObjectsUsingSDKs). For more information about Amazon S3 multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). This example shows how to copy an Amazon S3 object that is larger than 5 GB from one S3 bucket to another using the SDK for .NET multipart upload API. ``` using Amazon; using Amazon.S3; using Amazon.S3.Model; using System; using System.Collections.Generic; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class CopyObjectUsingMPUapiTest { private const string sourceBucket = "*** provide the name of the bucket with source object ***"; private const string targetBucket = "*** provide the name of the bucket to copy the object to ***"; private const string sourceObjectKey = "*** provide the name of object to copy ***"; private const string targetObjectKey = "*** provide the name of the object copy ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; public static void Main() { s3Client = new AmazonS3Client(bucketRegion); Console.WriteLine("Copying an object"); MPUCopyObjectAsync().Wait(); } private static async Task MPUCopyObjectAsync() { // Create a list to store the upload part responses. List uploadResponses = new List(); List copyResponses = new List(); // Setup information required to initiate the multipart upload. InitiateMultipartUploadRequest initiateRequest = new InitiateMultipartUploadRequest { BucketName = targetBucket, Key = targetObjectKey }; // Initiate the upload. InitiateMultipartUploadResponse initResponse = await s3Client.InitiateMultipartUploadAsync(initiateRequest); // Save the upload ID. String uploadId = initResponse.UploadId; try { // Get the size of the object. GetObjectMetadataRequest metadataRequest = new GetObjectMetadataRequest { BucketName = sourceBucket, Key = sourceObjectKey }; GetObjectMetadataResponse metadataResponse = await s3Client.GetObjectMetadataAsync(metadataRequest); long objectSize = metadataResponse.ContentLength; // Length in bytes. // Copy the parts. long partSize = 5 * (long)Math.Pow(2, 20); // Part size is 5 MB. long bytePosition = 0; for (int i = 1; bytePosition < objectSize; i++) { CopyPartRequest copyRequest = new CopyPartRequest { DestinationBucket = targetBucket, DestinationKey = targetObjectKey, SourceBucket = sourceBucket, SourceKey = sourceObjectKey, UploadId = uploadId, FirstByte = bytePosition, LastByte = bytePosition + partSize - 1 >= objectSize ? objectSize - 1 : bytePosition + partSize - 1, PartNumber = i }; copyResponses.Add(await s3Client.CopyPartAsync(copyRequest)); bytePosition += partSize; } // Set up to complete the copy. CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest { BucketName = targetBucket, Key = targetObjectKey, UploadId = initResponse.UploadId }; completeRequest.AddPartETags(copyResponses); // Complete the copy. CompleteMultipartUploadResponse completeUploadResponse = await s3Client.CompleteMultipartUploadAsync(completeRequest); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } } } } ``` ------ # Tutorial: Upload an object through multipart upload and verify its data integrity Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation. For more information about multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). For limits related to multipart uploads, see [Amazon S3 multipart upload limits](qfacts.md). You can use checksums to verify that assets are not altered when they are copied. Performing a checksum consists of using an algorithm to iterate sequentially over every byte in a file. Amazon S3 offers multiple checksum options for checking the integrity of data. We recommend that you perform these integrity checks as a durability best practice and to confirm that every byte is transferred without alteration. Amazon S3 also supports the following algorithms: SHA-1, SHA-256, CRC32, and CRC32C. Amazon S3 uses one or more of these algorithms to compute an additional checksum value and store it as part of the object metadata. For more information about checksums, see [Checking object integrity in Amazon S3](checking-object-integrity.md). **Objective** In this tutorial, you will learn how to upload an object to Amazon S3 by using a multipart upload and an additional SHA-256 checksum through the AWS Command Line Interface (AWS CLI). You’ll also learn how to check the object’s data integrity by calculating the MD5 hash and SHA-256 checksum of the uploaded object. **Topics** + [ ## Prerequisites ](#mpu-prerequisites) + [ ## Step 1: Create a large file ](#create-large-file-step1) + [ ## Step 2: Split the file into multiple files ](#split-large-file-step2) + [ ## Step 3: Create the multipart upload with an additional checksum ](#create-multipart-upload-step3) + [ ## Step 4: Upload the parts of your multipart upload ](#upload-parts-step4) + [ ## Step 5: List all the parts of your multipart upload ](#list-parts-step5) + [ ## Step 6: Complete the multipart upload ](#complete-multipart-upload-step6) + [ ## Step 7: Confirm that the object is uploaded to your bucket ](#confirm-upload-step7) + [ ## Step 8: Verify object integrity with an MD5 checksum ](#verify-object-integrity-step8) + [ ## Step 9: Verify object integrity with an additional checksum ](#verify-object-integrity-sha256-step9) + [ ## Step 10: Clean up your resources ](#clean-up-step10) ## Prerequisites + Before you start this tutorial, make sure that you have access to an Amazon S3 bucket that you can upload to. For more information, see [Creating a general purpose bucket](create-bucket-overview.md). + You must have the AWS CLI installed and configured. If you don’t have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. + Alternatively, you can run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. ## Step 1: Create a large file If you already have a file ready for upload, you can use the file for this tutorial. Otherwise, create a 15 MB file using the following steps. For limits related to multipart uploads, see [Amazon S3 multipart upload limits](qfacts.md). **To create a large file** Use one of the following commands to create your file, depending on which operating system you're using. **Linux or macOS** To create a 15 MB file, open your local terminal and run the following command: ``` dd if=/dev/urandom of=census-data.bin bs=1M count=15 ``` This command creates a file named `census-data.bin` filled with random bytes, with a size of 15 MB. **Windows** To create a 15 MB file, open your local terminal and run the following command: ``` fsutil file createnew census-data.bin 15728640 ``` This command creates a file named `census-data.bin` with a size of 15 MB of arbitrary data (15728640 bytes). ## Step 2: Split the file into multiple files To perform the multipart upload, you have to split your large file into smaller parts. You can then upload the smaller parts by using the multipart upload process. This step demonstrates how to split the large file created in [Step 1](#create-large-file-step1) into smaller parts. The following example uses a 15 MB file named `census-data.bin`. **To split a large file into parts** **Linux or macOS** To divide the large file into 5 MB parts, use the `split` command. Open your terminal and run the following: ``` split -b 5M -d census-data.bin census-part ``` This command splits `census-data.bin` into 5 MB parts named `census-part**`, where `**` is a numeric suffix starting from `00`. **Windows** To split the large file, use PowerShell. Open [Powershell](https://learn.microsoft.com/en-us/powershell/), and run the following script: ``` $inputFile = "census-data.bin" $outputFilePrefix = "census-part" $chunkSize = 5MB $fs = [System.IO.File]::OpenRead($inputFile) $buffer = New-Object byte[] $chunkSize $fileNumber = 0 while ($fs.Position -lt $fs.Length) { $bytesRead = $fs.Read($buffer, 0, $chunkSize) $outputFile = "{0}{1:D2}" -f $outputFilePrefix, $fileNumber $fileStream = [System.IO.File]::Create($outputFile) $fileStream.Write($buffer, 0, $bytesRead) $fileStream.Close() $fileNumber++ } $fs.Close() ``` This PowerShell script reads the large file in chunks of 5 MB and writes each chunk to a new file with a numeric suffix. After running the appropriate command, you should see the parts in the directory where you executed the command. Each part will have a suffix corresponding to its part number, for example: ``` census-part00 census-part01 census-part02 ``` ## Step 3: Create the multipart upload with an additional checksum To begin the multipart upload process, you need to create the multipart upload request. This step involves initiating the multipart upload and specifying an additional checksum for data integrity. The following example uses the SHA-256 checksum. If you want to provide any metadata describing the object being uploaded, you must provide it in the request to initiate the multipart upload. **Note** In this step and subsequent steps, this tutorial uses the SHA-256 additional algorithm. You might optionally use another additional checksum for these steps, such as CRC32, CRC32C, or SHA-1. If you use a different algorithm, you must use it throughout the tutorial steps. **To start the multipart upload** In your terminal, use the following `create-multipart-upload` command to start a multipart upload for your bucket. Replace `amzn-s3-demo-bucket1` with your actual bucket name. Also, replace the `census_data_file` with your chosen file name. This file name becomes the object key when the upload completes. ``` aws s3api create-multipart-upload --bucket amzn-s3-demo-bucket1 --key 'census_data_file' --checksum-algorithm sha256 ``` If your request succeeds, you'll see JSON output like the following: ``` { "ServerSideEncryption": "AES256", "ChecksumAlgorithm": "SHA256", "Bucket": "amzn-s3-demo-bucket1", "Key": "census_data_file", "UploadId": "cNV6KCSNANFZapz1LUGPC5XwUVi1n6yUoIeSP138sNOKPeMhpKQRrbT9k0ePmgoOTCj9K83T4e2Gb5hQvNoNpCKqyb8m3.oyYgQNZD6FNJLBZluOIUyRE.qM5yhDTdhz" } ``` **Note** When you send a request to initiate a multipart upload, Amazon S3 returns a response with an upload ID, which is a unique identifier for your multipart upload. You must include this upload ID whenever you upload parts, list the parts, complete an upload, or stop an upload. You'll need to use the `UploadId`, `Key`, and `Bucket` values for later steps, so make sure to save these. Also, if you’re using multipart upload with additional checksums, the part numbers must be consecutive. If you use nonconsecutive part numbers, the `complete-multipart-upload` request can result in an HTTP `500 Internal Server Error`. ## Step 4: Upload the parts of your multipart upload In this step, you will upload the parts of your multipart upload to your S3 bucket. Use the `upload-part` command to upload each part individually. This process requires specifying the upload ID, the part number, and the file to be uploaded for each part. **To upload the parts** 1. When uploading a part, in addition to the upload ID, you must specify a part number by using the `--part-number` argument. You can choose any part number between 1 and 10,000. A part number uniquely identifies a part and its position in the object you are uploading. The part number that you choose must be in a consecutive sequence (for example, it can be 1, 2, or 3). If you upload a new part using the same part number as a previously uploaded part, the previously uploaded part is overwritten. 1. Use the `upload-part` command to upload each part of your multipart upload. The `--upload-id` is the same as it was in the output created by the `create-multipart-upload` command in [Step 3](#create-multipart-upload-step3). To upload the first part of your data, use the following command: ``` aws s3api upload-part --bucket amzn-s3-demo-bucket1 --key 'census_data_file' --part-number 1 --body census-part00 --upload-id "cNV6KCSNANFZapz1LUGPC5XwUVi1n6yUoIeSP138sNOKPeMhpKQRrbT9k0ePmgoOTCj9K83T4e2Gb5hQvNoNpCKqyb8m3.oyYgQNZD6FNJLBZluOIUyRE.qM5yhDTdhz" --checksum-algorithm SHA256 ``` Upon completion of each `upload-part` command, you should see output like the following example: ``` { "ServerSideEncryption": "AES256", "ETag": "\"e611693805e812ef37f96c9937605e69\"", "ChecksumSHA256": "QLl8R4i4+SaJlrl8ZIcutc5TbZtwt2NwB8lTXkd3GH0=" } ``` 1. For subsequent parts, increment the part number accordingly: ``` aws s3api upload-part --bucket amzn-s3-demo-bucket1 --key 'census_data_file' --part-number --body --upload-id "" --checksum-algorithm SHA256 ``` For example, use the following command to upload the second part: ``` aws s3api upload-part --bucket amzn-s3-demo-bucket1 --key 'census_data_file' --part-number 2 --body census-part01 --upload-id "cNV6KCSNANFZapz1LUGPC5XwUVi1n6yUoIeSP138sNOKPeMhpKQRrbT9k0ePmgoOTCj9K83T4e2Gb5hQvNoNpCKqyb8m3.oyYgQNZD6FNJLBZluOIUyRE.qM5yhDTdhz" --checksum-algorithm SHA256 ``` Amazon S3 returns an entity tag (ETag) and additional checksums for each uploaded part as a header in the response. 1. Continue using the `upload-part` command until you have uploaded all the parts of your object. ## Step 5: List all the parts of your multipart upload To complete the multipart upload, you will need a list of all the parts that have been uploaded for that specific multipart upload. The output from the `list-parts` command provides information such as bucket name, key, upload ID, part number, ETag, additional checksums, and more. It’s helpful to save this output in a file so that you can use it for the next step when completing the multipart upload process. You can create a JSON output file called `parts.json` by using the following method. **To create a file that lists all of the parts** 1. To generate a JSON file with the details of all the uploaded parts, use the following `list-parts` command. Replace ***amzn-s3-demo-bucket1*** with your actual bucket name and **** with the upload ID that you received in [Step 3](#create-multipart-upload-step3). For more information on the `list-parts` command, see [https://docs.aws.amazon.com/cli/latest/reference/s3api/list-parts.html](https://docs.aws.amazon.com/cli/latest/reference/s3api/list-parts.html) in the *AWS Command Line Interface User Guide*. ``` aws s3api list-parts --bucket amzn-s3-demo-bucket1 --key 'census_data_file' --upload-id --query '{Parts: Parts[*].{PartNumber: PartNumber, ETag: ETag, ChecksumSHA256: ChecksumSHA256}}' --output json > parts.json ``` A new file called `parts.json` is generated. The file contains the JSON formatted information for all of your uploaded parts. The `parts.json` file includes essential information for each part of your multipart upload, such as the part numbers and their corresponding ETag values, which are necessary for completing the multipart upload process. 1. Open `parts.json` by using any text editor or through the terminal. Here’s the example output: ``` { "Parts": [ { "PartNumber": 1, "ETag": "\"3c3097f89e2a2fece47ac54b243c9d97\"", "ChecksumSHA256": "fTPVHfyNHdv5VkR4S3EewdyioXECv7JBxN+d4FXYYTw=" }, { "PartNumber": 2, "ETag": "\"03c71cc160261b20ab74f6d2c476b450\"", "ChecksumSHA256": "VDWTa8enjOvULBAO3W2a6C+5/7ZnNjrnLApa1QVc3FE=" }, { "PartNumber": 3, "ETag": "\"81ae0937404429a97967dffa7eb4affb\"", "ChecksumSHA256": "cVVkXehUlzcwrBrXgPIM+EKQXPUvWist8mlUTCs4bg8=" } ] } ``` ## Step 6: Complete the multipart upload After uploading all parts of your multipart upload and listing them, the final step is to complete the multipart upload. This step merges all the uploaded parts into a single object in your S3 bucket. **Note** You can calculate the object checksum before calling `complete-multipart-upload` by including `--checksum-sha256` in your request. If the checksums don't match, Amazon S3 fails the request. For more information, see [https://docs.aws.amazon.com/cli/latest/reference/s3api/complete-multipart-upload.html](https://docs.aws.amazon.com/cli/latest/reference/s3api/complete-multipart-upload.html) in the *AWS Command Line Interface User Guide*. **To complete the multipart upload** To finalize the multipart upload, use the `complete-multipart-upload` command. This command requires the `parts.json` file created in [Step 5](#list-parts-step5), your bucket name, and the upload ID. Replace **<*amzn-s3-demo-bucket1*>** with your bucket name and **** with the upload ID of `parts.json`. ``` aws s3api complete-multipart-upload --multipart-upload file://parts.json --bucket amzn-s3-demo-bucket1 --key 'census_data_file' --upload-id ``` Here’s the example output: ``` { "ServerSideEncryption": "AES256", "Location": "https://amzn-s3-demo-bucket1.s3.us-east-2.amazonaws.com/census_data_file", "Bucket": "amzn-s3-demo-bucket1", "Key": "census_data_file", "ETag": "\"f453c6dccca969c457efdf9b1361e291-3\"", "ChecksumSHA256": "aI8EoktCdotjU8Bq46DrPCxQCGuGcPIhJ51noWs6hvk=-3" } ``` **Note** Don't delete the individual part files yet. You will need the individual parts so that you can perform checksums on them to verify the integrity of the merged-together object. ## Step 7: Confirm that the object is uploaded to your bucket After completing the multipart upload, you can verify that the object has been successfully uploaded to your S3 bucket. To list the objects in your bucket and confirm the presence of your newly uploaded file, use the `list-objects-v2` command **To list the uploaded object** To list the objects in your, use the `list-objects-v2` command bucket. Replace ***amzn-s3-demo-bucket1*** with your actual bucket name: ``` aws s3api list-objects-v2 --bucket amzn-s3-demo-bucket1 ``` This command returns a list of objects in your bucket. Look for your uploaded file (for example, `census_data_file`) in the list of objects. For more information, see the [Examples](https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html) section for the `list-objects-v2` command in the *AWS Command Line Interface User Guide*. ## Step 8: Verify object integrity with an MD5 checksum When you upload an object, you can specify a checksum algorithm for Amazon S3 to use. By default, Amazon S3 stores the MD5 digest of bytes as the object’s ETag. For multipart uploads, the ETag is not the checksum for the entire object, but rather a composite of checksums for each individual part. **To verify object integrity by using an MD5 checksum** 1. To retrieve the ETag of the uploaded object, perform a `head-object` request: ``` aws s3api head-object --bucket amzn-s3-demo-bucket1 --key census_data_file ``` Here’s the example output: ``` { "AcceptRanges": "bytes", "LastModified": "2024-07-26T19:04:13+00:00", "ContentLength": 16106127360, "ETag": "\"f453c6dccca969c457efdf9b1361e291-3\"", "ContentType": "binary/octet-stream", "ServerSideEncryption": "AES256", "Metadata": {} } ``` This ETag has "-3" appended to the end. This indicates that the object was uploaded in three parts using multipart upload. 1. Next, calculate the MD5 checksum of each part using the `md5sum` command. Make sure that you provide the correct path to your part files: ``` md5sum census-part* ``` Here’s the example output: ``` e611693805e812ef37f96c9937605e69 census-part00 63d2d5da159178785bfd6b6a5c635854 census-part01 95b87c7db852451bb38b3b44a4e6d310 census-part02 ``` 1. For this step, manually combine the MD5 hashes into one string. Then, run the following command to convert the string to binary and calculate the MD5 checksum of the binary value: ``` echo "e611693805e812ef37f96c9937605e6963d2d5da159178785bfd6b6a5c63585495b87c7db852451bb38b3b44a4e6d310" | xxd -r -p | md5sum ``` Here’s the example output: ``` f453c6dccca969c457efdf9b1361e291 - ``` This hash value should match the hash value of the original ETag value in [Step 1](#create-large-file-step1), which validates the integrity of the `census_data_file` object. When you instruct Amazon S3 to use additional checksums, Amazon S3 calculates the checksum value for each part and stores the values. If you want to retrieve the checksum values for individual parts of multipart uploads that are still in progress, you can use `list-parts`. For more information about how checksums work with multipart upload objects, see [Checking object integrity in Amazon S3](checking-object-integrity.md). ## Step 9: Verify object integrity with an additional checksum In this step, this tutorial uses SHA-256 as an additional checksum to validate object integrity. If you’ve used a different additional checksum, use that checksum value instead. **To verify object integrity with SHA256** 1. Run the following command in your terminal, including the `--checksum-mode enabled` argument, to display the `ChecksumSHA256` value of your object: ``` aws s3api head-object --bucket amzn-s3-demo-bucket1 --key census_data_file --checksum-mode enabled ``` Here’s the example output: ``` { "AcceptRanges": "bytes", "LastModified": "2024-07-26T19:04:13+00:00", "ContentLength": 16106127360, "ChecksumSHA256": "aI8EoktCdotjU8Bq46DrPCxQCGuGcPIhJ51noWs6hvk=-3", "ETag": "\"f453c6dccca969c457efdf9b1361e291-3\"", "ContentType": "binary/octet-stream", "ServerSideEncryption": "AES256", "Metadata": {} } ``` 1. Use the following commands to decode the `ChecksumSHA256` values for the individual parts into base64 and save them into a binary file called `outfile`. These values can be found in your `parts.json` file. Replace the example base64 strings with your actual `ChecksumSHA256` values. ``` echo "QLl8R4i4+SaJlrl8ZIcutc5TbZtwt2NwB8lTXkd3GH0=" | base64 --decode >> outfile echo "xCdgs1K5Bm4jWETYw/CmGYr+m6O2DcGfpckx5NVokvE=" | base64 --decode >> outfile echo "f5wsfsa5bB+yXuwzqG1Bst91uYneqGD3CCidpb54mAo=" | base64 --decode >> outfile ``` 1. Run the following command to calculate the SHA256 checksum of the `outfile`: ``` sha256sum outfile ``` Here’s the example output: ``` 688f04a24b42768b6353c06ae3a0eb3c2c50086b8670f221279d67a16b3a86f9 outfile ``` In the next step, take the hash value and convert it into a binary value. This binary value should match the `ChecksumSHA256` value from [Step 1](#create-large-file-step1). 1. Convert the SHA256 checksum from [Step 3](#create-multipart-upload-step3) into binary, and then encode it to base64 to verify that it matches the `ChecksumSHA256` value from [Step 1](#create-large-file-step1): ``` echo "688f04a24b42768b6353c06ae3a0eb3c2c50086b8670f221279d67a16b3a86f9" | xxd -r -p | base64 ``` Here’s the example output: ``` aI8EoktCdotjU8Bq46DrPCxQCGuGcPIhJ51noWs6hvk= ``` This output should confirm that the base64 output matches the `ChecksumSHA256` value from the `head-object` command output. If the output matches the checksum value, then the object is valid. **Important** When you instruct Amazon S3 to use additional checksums, Amazon S3 calculates the checksum values for each part and stores these values. If you want to retrieve the checksum values for individual parts of multipart uploads that are still in progress, you can use the `list-parts` command. ## Step 10: Clean up your resources If you want to clean up the files created in this tutorial, use the following method. For instructions on deleting the files uploaded to your S3 bucket, see [Deleting Amazon S3 objects](DeletingObjects.md). **Delete local files created in [Step 1](#create-large-file-step1):** To remove the files that you created for your multipart upload, run the following command from your working directory: ``` rm census-data.bin census-part* outfile parts.json ``` # Amazon S3 multipart upload limits Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation. For more information about multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). The following table provides multipart upload core specifications. These include maximum object size, maximum number of parts, maximum part size, and more. There is no minimum size limit on the last part of your multipart upload. | Item | Specification | | --- | --- | | Maximum object size | 48.8 TiB | | Maximum number of parts per upload | 10,000 | | Part numbers | 1 to 10,000 (inclusive) | | Part size | 5 MiB to 5 GiB. There is no minimum size limit on the last part of your multipart upload. | | Maximum number of parts returned for a list parts request | 1000 | | Maximum number of multipart uploads returned in a list multipart uploads request | 1000 | # Add preconditions to S3 operations with conditional requests You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing. Conditional reads are supported for `GET`, `HEAD`, and `COPY` requests. You can add preconditions to return or copy an object based on its Entity tag (ETag) or last modified date. This can limit an S3 operation to objects updated since a specified date. You can also limit an S3 operation to a specific ETag. This could ensure you only return or copy a specific object version. For more information about the object metadata, see [Working with object metadata](UsingMetadata.md). Conditional writes can ensure there is no existing object with the same key name in your bucket during `PUT` operations. This prevents overwriting of existing objects with identical key names. Similarly, you can use conditional writes to check if an object's ETag is unchanged before updating the object. This prevents unintentional overwrites on an object without knowing the state of its content. You can use conditional writes for [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html), [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html), or [CopyObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) requests. For more information about key names, see [Naming Amazon S3 objects](object-keys.md). Conditional deletes evaluate if your object exists or is unchanged before deleting it. You can perform conditional deletes using the `DeleteObject` or `DeleteObjects` APIs in general purpose and directory buckets. For more information on conditional deletes, see [How to perform conditional deletes](conditional-deletes.md). There is no additional charge for conditional reads, conditional writes or conditional deletes. You are only charged existing rates for the applicable requests, including for failed requests. For information about Amazon S3 features and pricing, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing). **Topics** + [ # How to retrieve or copy objects based on metadata with conditional reads ](conditional-reads.md) + [ # How to prevent object overwrites with conditional writes ](conditional-writes.md) + [ # How to perform conditional deletes ](conditional-deletes.md) # How to retrieve or copy objects based on metadata with conditional reads With conditional reads, you can add an additional header to your read request in order to add preconditions to your S3 operation. If these preconditions are not met the read request will fail. You can use conditional reads on `GET`, `HEAD`, or `COPY` requests to only return an object based on its metadata. When you upload an object, Amazon S3 creates system controlled metadata that can only be modified by S3. Entity tags (ETags) and Last-Modified are examples of system controlled metadata. An object's ETag is a string representing a specific version of an object. Last-Modified date is metadata representing an object's creation date or the last modified date, whichever is the latest. With conditional reads, you can return an object based on the object's ETag or Last-Modified date. You can specify an ETag value with your request and return the object only if the ETag value matches. This can ensure you only return or copy a specific version of an object. You can specify a Last-Modified value with your read request and return an object only if that object has been modified since a date you provide. ## Supported APIs The following S3 APIs support using conditional reads: + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) You can use the following headers to return an object dependent on the entity tag (ETag) or last modified date. For more information about object metadata such as ETags and Last-Modified, see [System-defined object metadata](UsingMetadata.md#SysMetadata). **[https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)** + `If-Match` — Return the object only if its ETag matches the one provided. + `If-Modified-Since` — Return the object only if it has been modified since the time specified. + `If-None-Match` — Return the object only if its ETag does not matches the one provided. + `If-Unmodified-Since` — Return the object only if it has not been modified since the time specified. For more information about these headers, errors returned, and the order S3 handles multiple conditional headers in a single request, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) in the Amazon Simple Storage Service API Reference. **[https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html)** + `If-Match` — Return the object only if its ETag matches the one provided. + `If-Modified-Since` — Return the object only if it has been modified since the time specified. + `If-None-Match` — Return the object only if its ETag does not matches the one provided. + `If-Unmodified-Since` — Return the object only if it has not been modified since the time specified. For more information about these headers, errors returned, and the order S3 handles multiple conditional headers in a single request, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) in the Amazon Simple Storage Service API Reference. **[https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html)** + `x-amz-copy-source-if-match` — Copies the source object only if its ETag matches the one provided. + `x-amz-copy-source-if-modified-since` — Copies the source object only if it has been modified since the time specified. + `x-amz-copy-source-if-none-match` — Copies the source object only if its ETag does not matches the one provided. + `x-amz-copy-source-if-unmodified-since` — Copies the source object only if it has not been modified since the time specified. + `If-Match` — Copies the object only if its ETag matches the one provided. `If-Match` expects the ETag value as a string. + `If-None-Match` — Copies the object only if its ETag does not match the one provided. `If-None-Match` expects the '\$1' (asterisk) character. For more information about these headers, errors returned, and the order S3 handles multiple conditional headers in a single request, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) in the Amazon Simple Storage Service API Reference. # How to prevent object overwrites with conditional writes By using conditional writes, you can add an additional header to your `WRITE` requests to specify preconditions for your Amazon S3 operation. To conditionally write objects, add the HTTP `If-None-Match` or `If-Match` header. The `If-None-Match` header prevents overwrites of existing data by validating that there's not an object with the same key name already in your bucket. Alternatively, you can add the `If-Match` header to check an object's entity tag (ETag) before writing an object. With this header, Amazon S3 compares the provided ETag value with the ETag value of the object in S3. If the ETag values don't match, the operation fails. Bucket owners can use bucket policies to enforce conditional writes for uploaded objects. For more information, see [Enforce conditional writes on Amazon S3 buckets](conditional-writes-enforce.md). **Note** To use conditional writes, you must use AWS Signature Version 4 to sign the request. **Topics** + [ ## How to prevent object overwrites based on key names ](#conditional-write-key-names) + [ ## How to prevent overwrites if the object has changed ](#conditional-write-etags) + [ ## Conditional write behavior ](#conditional-error-response) + [ ## Conditional write scenarios ](#conditional-write-scenarios) + [ # Enforce conditional writes on Amazon S3 buckets ](conditional-writes-enforce.md) ## How to prevent object overwrites based on key names You can use the HTTP `If-None-Match` conditional header to check whether an object already exists in the specified bucket based on its key name before creating it or copying it to the destination bucket. Conditional writes with the HTTP `If-None-Match` header check for the existence of an object during the `WRITE` operation. If an identical key name is found in the bucket, the operation fails. Without the HTTP `If-None-Match` header, if you upload or copy an object with an identical key name in an unversioned or version-suspended bucket, the object is overwritten. For more information about using key names, see [Naming Amazon S3 objects](object-keys.md). **Note** The HTTP `If-None-Match` header only applies to the current version of an object in a version bucket. To perform conditional writes with the HTTP `If-None-Match` header you must have the `s3:PutObject` permission. This enables the caller to check for the presence of objects in the bucket. The `If-None-Match` header expects the \$1 (asterisk) value. You can use the `If-None-Match` header with the following APIs: + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) ### Conditional Put Using the AWS CLI The following `put-object` example command attempts to perform a conditional write for an object with the key name `dir-1/my_images.tar.bz2`. ``` aws s3api put-object --bucket amzn-s3-demo-bucket --key dir-1/my_images.tar.bz2 --body my_images.tar.bz2 --if-none-match "*" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-object.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ### Conditional Copy Using the AWS CLI The following `copy-object` example command attempts to copy an object to a destination bucket with a conditional write for an object with the key name `dir-1/my_images.tar.bz2`. ``` aws s3api copy-object --copy-source amzn-s3-demo-bucket/key --key dir-1/my_images.tar.bz2 --bucket amzn-s3-demo-bucket2 --if-none-match "*" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/copy-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/copy-object.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ### Conditional multipart upload Using the AWS CLI The following `complete-multipart-upload` example command attempts to complete a multipart upload with a conditional write for an object with the key name `dir-1/my_images.tar.bz2`. In this example, the file:// prefix is used to load the JSON structure from a file in the local folder named `mpustruct` which list of all the parts that have been uploaded for tnis specific multipart upload. ``` aws s3api complete-multipart-upload --multipart-upload file://mpustruct --bucket amzn-s3-demo-bucket --key dir-1/my_images.tar.bz2 --upload-id upload-id --if-none-match "*" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/complete-multipart-upload.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/complete-multipart-upload.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ## How to prevent overwrites if the object has changed An object's ETag is a string that's unique to the object and reflects a change to the object's content. You can use the `If-Match` header to compare the ETag value of an object in an Amazon S3 bucket with one that you provide during the `WRITE` operation. If the ETag values don't match, the operation fails. For more information about ETags, see [Using Content-MD5 and the ETag to verify uploaded objects](checking-object-integrity-upload.md#checking-object-integrity-etag-and-md5). To perform conditional writes with an HTTP `If-Match` header you must have the `s3:PutObject` and `s3:GetObject` permissions. This enables the caller to check the ETag and verify the state of the objects in the bucket. The `If-Match` header expects the ETag value as a string. You can use the `If-Match` header with the following APIs: + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) ### Conditional Put Using the AWS CLI The following `put-object` example command attempts to perform a conditional write with the provided ETag value `6805f2cfc46c0f04559748bb039d69ae`. ``` aws s3api put-object --bucket amzn-s3-demo-bucket --key dir-1/my_images.tar.bz2 --body my_images.tar.bz2 --if-match "6805f2cfc46c0f04559748bb039d69ae" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-object.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ### Conditional Copy Using the AWS CLI The following `copy-object` example command attempts to perform a conditional write with the provided ETag value `6805f2cfc46c0f04559748bb039d69ae`. ``` aws s3api copy-object --copy-source amzn-s3-demo-bucket/key --key dir-1/my_images.tar.bz2 --bucket amzn-s3-demo-bucket2 --if-match "6805f2cfc46c0f04559748bb039d69ae" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/copy-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/copy-object.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ### Conditional multipart upload Using the AWS CLI The following `complete-multipart-upload` example command attempts to complete a multipart upload with a conditional write using the provided ETag value `6805f2cfc46c0f04559748bb039d69ae`. In this example, the file:// prefix is used to load the JSON structure from a file in the local folder named `mpustruct` which list of all the parts that have been uploaded for tnis specific multipart upload. ``` aws s3api complete-multipart-upload --multipart-upload file://mpustruct --bucket amzn-s3-demo-bucket --key dir-1/my_images.tar.bz2 --upload-id upload-id --if-match "6805f2cfc46c0f04559748bb039d69ae" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/complete-multipart-upload.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/complete-multipart-upload.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ## Conditional write behavior **Conditional writesor copies with `If-None-Match` header** Conditional writes with the `If-None-Match` header evaluate against existing objects in a bucket. If there's no existing object with the same key name in the bucket, the write operation succeeds, resulting in a `200 OK` response. If there's an existing object, the write operation fails, resulting in a `412 Precondition Failed` response. For buckets with versioning enabled, if there's no current object version with the same name, or if the current object version is a delete marker, the write operation succeeds. Otherwise, it results in a failed write operation with a `412 Precondition Failed` response. If multiple conditional writes or copies occur for the same object name, the first write operation to finish succeeds. Amazon S3 then fails subsequent writes with a `412 Precondition Failed` response. You can also receive a `409 Conflict` response in the case of concurrent requests if a delete request to an object succeeds before a conditional write operation on that object completes. When using conditional writes with `PutObject`, uploads may be retried after receiving a `409 Conflict` error. When using `CompleteMultipartUpload`, the entire multipart upload must be re-initiated with `CreateMultipartUpload` to upload the object again after receiving a `409 Conflict` error. **Conditional writes or copies with `If-Match` header** The `If-Match` header evaluates against existing objects in a bucket. If there's an existing object with the same key name and matching ETag, the write operation succeeds, resulting in a `200 OK` response. If the ETag doesn't match, the write operation fails with a `412 Precondition Failed` response. You can also receive a `409 Conflict` response in the case of concurrent requests. You will receive a `404 Not Found` response if a concurrent delete request to an object succeeds before a conditional write operation on that object completes, as the object key no longer exists. You should reupload the object when you receive a `404 Not Found` response. If there's no current object version with the same name, or if the current object version is a delete marker, the operation fails with a `404 Not Found` error. ## Conditional write scenarios Consider the following scenarios where two clients are running operations on the same bucket. **Conditional writes during multipart uploads** Conditional writes do not consider any in-progress multipart uploads requests since those are not yet fully written objects. Consider the following example where Client 1 is uploading an object using multipart upload. During the multipart upload, Client 2 is able to successfully write the same object with the conditional write operation. Subsequently, when Client 1 tries to complete the multipart upload using a conditional write the upload fails. **Note** This scenario will result in a `412 Precondition Failed` response for both `If-None-Match` and `If-Match` headers. ![\[An example of two clients writing items with the same key name. One with UploadPart for MPU and one with PutObject and a conditional write. The CompleteMultipartUpload operation, which starts after, fails.\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/images/conwrite_put_mpu.png) **Concurrent deletes during multipart uploads** If a delete request succeeds before a conditional write request can complete, Amazon S3 returns a `409 Conflict` or `404 Not Found` response for the write operation. This is because the delete request that was initiated earlier takes precedence over the conditional write operation. In such cases, you must initiate a new multipart upload. **Note** This scenario will result in a `409 Conflict` response for an `If-None-Match` header and a `404 Not Found` response for an `If-Match` header. ![\[An example of two clients, one using multipart upload and one sending a delete request after the MPU has started. The delete request finishes before the conditional write starts.\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/images/conwrite_delete_mpu.png) **Note** To minimize your storage costs, we recommend that you configure a lifecycle rule to delete incomplete multipart uploads after a specified number of days by using the `AbortIncompleteMultipartUpload` action. For more information about creating a lifecycle rule to delete incomplete multipart uploads, see [Configuring a bucket lifecycle configuration to delete incomplete multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html). # Enforce conditional writes on Amazon S3 buckets By using Amazon S3 bucket policies, you can enforce conditional writes for object uploads in your general purpose buckets. A bucket policy is a resource-based policy that you can use to grant access permissions to your Amazon S3 bucket and the objects in it. Only the bucket owner can associate a policy with a bucket. For more information about bucket policies, see [Bucket policies for Amazon S3](bucket-policies.md). You can use the condition keys `s3:if-match` or `s3:if-none-match` as the optional `Condition` element or `Condition` block to specify when a policy is in effect. For multipart uploads you must specify the `s3:ObjectCreationOperation` condition key to exempt the `CreateMultipartUpload`, `UploadPart`, and `UploadPartCopy` operations, as these APIs don't accept conditional headers. For more information about using conditions in bucket policies, see [Bucket policy examples using condition keys](amazon-s3-policy-keys.md). **Note** If you use a bucket policy to enforce conditional writes, you can't perform copy operations to the bucket or prefix specified in your bucket policy. `CopyObject` requests without an `If-None-Match` or `If-Match` HTTP header fail with a `403 Access Denied` error. `CopyObject` requests made with those HTTP headers fail with a `501 Not Implemented` response. The following examples show how to use conditions in a bucket policy to force clients to use the `If-None-Match` or `If-Match` HTTP header. **Topics** + [ ## Example 1: Only allow object uploads using `PutObject` and `CompleteMultipartUpload` requests that include the `if-none-match` header ](#conditional-writes-enforce-ex1) + [ ## Example 2: Only allow object uploads using `PutObject` and `CompleteMultipartUpload` requests that include the `if-match` header ](#conditional-writes-enforce-ex2) + [ ## Example 3: Only allow object upload requests that includes the `if-none-match` or `if-match` header ](#conditional-writes-enforce-ex3) ## Example 1: Only allow object uploads using `PutObject` and `CompleteMultipartUpload` requests that include the `if-none-match` header This policy allows account 111122223333, user Alice, to write to the *amzn-s3-demo-bucket1* bucket if the request includes the `if-none-match` header, ensuring that the object key doesn't already exist in the bucket. All `PutObject` and `CompleteMultipartUpload` requests to the specified bucket must include the `if-none-match` header to succeed. Using this header, customers can write to this bucket only if the object key does not exist in the bucket. **Note** This policy also sets the `s3:ObjectCreationOperation` condition key which allows for multipart uploads using the `CreateMultipartUpload`, `UploadPart`, and `UploadPartCopy` APIs. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowConditionalPut", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*", "Condition": { "Null": { "s3:if-none-match": "false" } } }, { "Sid": "AllowConditionalPutwithMPUs", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*", "Condition": { "Bool": { "s3:ObjectCreationOperation": "false" } } } ] } ``` ------ ## Example 2: Only allow object uploads using `PutObject` and `CompleteMultipartUpload` requests that include the `if-match` header This policy allows account 111122223333, user Alice to write to *amzn-s3-demo-bucket1* only if the request includes the `if-match` header. This header compares the ETag value of an object in S3 with one you provide during the `WRITE` operation. If the ETag values do not match, the operation will fail. All `PutObject` and `CompleteMultipartUpload` requests to the specified bucket must include the `if-match` header to succeed. **Note** This policy also sets the `s3:ObjectCreationOperation` condition key which allows for multipart uploads using the `CreateMultipartUpload`, `UploadPart`, and `UploadPartCopy` APIs. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowPutObject", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*", }, { "Sid": "BlockNonConditionalObjectCreation", "Effect": "Deny", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*", "Condition": { "Null": { "s3:if-match": "true" }, "Bool": { "s3:ObjectCreationOperation": "true" } } }, { "Sid": "AllowGetObjectBecauseConditionalPutIfMatchETag", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*" } ] } ``` ## Example 3: Only allow object upload requests that includes the `if-none-match` or `if-match` header This policy allows account 111122223333, user Alice to write to *amzn-s3-demo-bucket1* if the requests include the `if-none-match` or `if-match` header. This allows Alice to upload an object if the key name does not exist in the bucket, or if the key name does exist, Alice can overwrite the object if the object ETag matches the ETag provided in the `PUT` request. **Note** This policy also sets the `s3:ObjectCreationOperation` condition key which allows for multipart uploads using the `CreateMultipartUpload`, `UploadPart`, and `UploadPartCopy` APIs. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": " AllowConditionalPutifAbsent", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*", "Condition": { "Null": { "s3:if-none-match": "false" } } }, { "Sid": "AllowConditionalPutIfMatchEtag", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*", "Condition": { "Null": { "s3:if-match": "false" } } }, { "Sid": "AllowConditionalObjectCreation", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*", "Condition": { "Bool": { "s3:ObjectCreationOperation": "false" } } }, { "Sid": " AllowGetObjectBecauseConditionalPutIfMatchETag", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket1/*" } ] } ``` ------ # How to perform conditional deletes You can use conditional deletes to evaluate if your object exists or is unchanged before deleting it. You can perform conditional deletes using the `DeleteObject` or `DeleteObjects` API operations in S3 general purpose and directory buckets. To get started, when making a conditional delete request, you can use the `HTTP If-Match` header with the precondition value `*` to check if object exists or the `If-Match` header with your provided `ETag` to check if the object has been modified. You can enforce conditional deletes at a general purpose bucket level using S3 bucket or Identity and Access Management (IAM) policies. For more information, see [Enforce conditional deletes on Amazon S3 buckets](conditional-delete-enforce.md). **Note** Conditional delete evaluations only apply to the current version of the object. **Topics** + [ ## How to check if your object has been modified before deleting it ](#conditional-deletes-etags) + [ ## How to check if your object exists before deleting it ](#conditional-delete) + [ # Enforce conditional deletes on Amazon S3 buckets ](conditional-delete-enforce.md) ## How to check if your object has been modified before deleting it With conditional deletes, you can protect your application from accidental deletions of objects. You can use the `HTTP If-Match` header with the `ETag` value to check if an object has been modified. If the `ETag` value of an object in an S3 bucket doesn’t match with the `ETag` that you provide during the delete operation, the operation fails. For conditionally deleting multiple objects using the `DeleteObjects` operation, you must provide the `ETag` value in the `ETag` element of the object in the XML request body. For more information, see [Using Content-MD5 and the ETag to verify uploaded objects](checking-object-integrity-upload.md#checking-object-integrity-etag-and-md5). **Note** To perform conditional deletes with the `If-Match` header with the `ETag` value, you must have the `s3:DeleteObject` and `s3:GetObject` permissions. The `If-Match` header with the `ETag` value evaluates against existing objects in a bucket. If there's an existing object with the same key name and matching `ETag`, the `DeleteObject` requests succeeds, and returns a `204 No content` response. If the `ETag` doesn't match, the delete operation fails with a `412 Precondition Failed` response. To conditionally delete multiple objects using the `DeleteObjects` operation, you can provide the `ETag` value in the `ETag` element of the object in the XML request body. If the request succeeds, the `DeleteObjects` operation responds with a `200 OK` and provides the status of each object in the response body. If the precondition succeeds, the response for that object will be captured in the `` element of the response body. If the precondition fails then the response for that object will be captured in the `` element of the response body. You can also receive a `409 Conflict` error response in the case of concurrent requests if a `DELETE` or `PUT` request to an object succeeds before a conditional delete operation on that object completes. You will receive a `404 Not Found` response if a concurrent delete request to an object succeeds before a conditional write operation on that object completes, as the object key no longer exists. You can use the `If-Match` header with the `ETag` value for the following APIs: + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) ### Using the AWS CLI The following `delete-object` example command attempts to perform a conditional delete with the provided ETag value `6805f2cfc46c0f04559748bb039d69al`. ``` aws s3api delete-object --bucket amzn-s3-demo-bucket --key dir-1/my_images.tar.bz2 --if-match "6805f2cfc46c0f04559748bb039d69al" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-object.html) in the *AWS CLI Command Reference*. The following `delete-objects` example command attempts to perform a conditional delete with the provided ETag value `6805f2cfc46c0f04559748bb039d69al`. ``` aws s3api delete-objects --bucket amzn-s3-demo-bucket --delete '{"Objects":[{"Key":"my_images.tar.bz2", "ETag": "6805f2cfc46c0f04559748bb039d69al"}]}' ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-objects.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-objects.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ## How to check if your object exists before deleting it You can use the `If-Match` header with the `*` value to check if the object exists before attempting to delete it. The `*` value signifies that the operation should only proceed if the object exists, regardless of whether it has been modified or not. Delete markers are special objects in versioned S3 general purpose buckets that indicate that an object has been deleted. They are placeholders that make the object appear deleted while preserving the previous versions. Therefore, when you use `If-Match:*` with a `DeleteObject` API, the operation will only succeed with a `204 No Content` if the object exists. If the latest version of the object is a delete marker, the object doesn't exist and the `DeleteObject` API will fail and return a `412 Precondition Failed` response. For more information about delete markers, see [Working with delete markers](DeleteMarker.md). To conditionally delete multiple objects using the `DeleteObjects` operation, you can provide the `*` in the `ETag` element of the object in the XML request body. If the precondition succeeds, the `DeleteObjects` operation responds with a `200 OK` and provides the status of each object in the response body. If the precondition succeeds, the response for that object will be captured in the `` element of the response body. If the precondition fails then the response for that object will be captured in the `` element of the response body. If the object doesn’t exist when evaluating either of the preconditions, S3 rejects the request and returns a `Not Found` error response. **Note** To perform conditional deletes with `If-Match:*`, you must have `s3:DeleteObject` permissions. You can use the `If-Match` header with the `*` value for the following APIs: + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) ### Using the AWS CLI The following `delete-object` example command attempts to perform a conditional delete for an object with the key name `my_images.tar.bz2` that has a value of `*` which represents any ETag. ``` aws s3api delete-object --bucket amzn-s3-demo-bucket --key dir-1/my_images.tar.bz2 --if-match "*" ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-object.html) in the *AWS CLI Command Reference*. The following `delete-objects` example command attempts to perform a conditional delete for an object with the key name `my_images.tar.bz2` that has a value of `*` which represents any ETag. ``` aws s3api delete-objects --bucket amzn-s3-demo-bucket --delete '{"Objects":[{"Key":"my_images.tar.bz2", "ETag": "*"}]}' ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-objects.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-objects.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. # Enforce conditional deletes on Amazon S3 buckets By using Amazon S3 bucket policies, you can enforce `If-Match`header with conditional deletes for objects in general purpose buckets. If the `If-Match` header doesn’t exist, the request will be denied with an `403 Access Denied`. A bucket policy is a resource-based policy that you can use to grant access permissions to your bucket and the objects in it. Only the bucket owner can associate a policy with a bucket. For more information about bucket policies, see [Bucket policies for Amazon S3](bucket-policies.md). The following examples show how to use conditions in a bucket policy to force clients to use the `If-Match` HTTP header. **Topics** + [ ## Example 1: Only allow conditional deletes using the `If-Match` header with the `ETag` value ](#conditional-writes-enforce-ex1) + [ ## Example 2: Only allow conditional deletes using the `If-Match` header with the `*` value ](#conditional-deletes-enforce-ex2) ## Example 1: Only allow conditional deletes using the `If-Match` header with the `ETag` value You can use this bucket policy to only allow conditional deletes using `DeleteObject` and `DeleteObjects` requests that include the `If-Match` header with the `ETag` value. The `Null` condition ensures the `If-Match` header is present, and the `s3:GetObject` permission is granted because conditional deletes with a specific ETag value require both `s3:DeleteObject` and `s3:GetObject` permissions. All non-conditional deletes would be denied and conditional deletes would pass. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowConditionalDeletes", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:DeleteObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*", "Condition": { "Null": { "s3:if-match": "false" } } }, { "Sid": "AllowGetObjectBecauseConditionalDeleteIfMatchETag", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*" } ] } ``` ## Example 2: Only allow conditional deletes using the `If-Match` header with the `*` value You can use this bucket policy to only allow conditional deletes using `DeleteObject` and `DeleteObjects` requests that include the `If-Match` header with the `*` value. The `Null` condition ensures the `If-Match` header is present. Because `s3:GetObject` is not granted, conditional deletes with a specific ETag value will fail – only `If-Match: *` (which checks object existence and requires only `s3:DeleteObject` permission) will succeed. All non-conditional deletes would be denied, and only `If-Match: *` conditional deletes would succeed. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowConditionalDeletes", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:user/Alice" }, "Action": "s3:DeleteObject", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*", "Condition": { "Null": { "s3:if-match": "false" } } } ] } ``` # Copying, moving, and renaming objects The `CopyObject` operation creates a copy of an object that's already stored in Amazon S3. You can create a copy of an object up to 5 GB in a single atomic operation. However, to copy an object that's larger than 5 GB, you must use a multipart upload using the AWS CLI or AWS SDKs. For more information, see [Copying an object using multipart upload](CopyingObjectsMPUapi.md). **Note** To maintain the performance benefit of an object you uploaded using multipart upload, you must copy the object using multipart upload using the AWS CLI or AWS SDK instead of the S3 console. For more information, see [Copying an object using multipart upload](CopyingObjectsMPUapi.md). Using the `CopyObject` operation, you can: + Create additional copies of objects. + Rename objects by copying them and deleting the original ones. + Copy or move objects from one bucket to another, including across AWS Regions (for example, from `us-west-1` to `eu-west-2`). When you move an object, Amazon S3 copies the object to the specified destination and then deletes the source object. **Note** Copying or moving objects across AWS Regions incurs bandwidth charges. For more information, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). + Change object metadata. Each Amazon S3 object has metadata. This metadata is a set of name-value pairs. You can set object metadata at the time you upload an object. After you upload the object, you can't modify the object metadata. The only way to modify object metadata is to make a copy of the object and set the metadata. To do so, in the copy operation, set the same object as the source and target. Some object metadata is system metadata and other is user-defined. You can control some of the system metadata. For example, you can control the storage class and the type of server-side encryption to use for the object. When you copy an object, user-controlled system metadata and user-defined metadata are also copied. Amazon S3 resets the system-controlled metadata. For example, when you copy an object, Amazon S3 resets the creation date of the copied object. You don't need to set any of these system-controlled metadata values in your copy request. When copying an object, you might decide to update some of the metadata values. For example, if your source object is configured to use S3 Standard storage, you might choose to use S3 Intelligent-Tiering for the object copy. You might also decide to alter some of the user-defined metadata values present on the source object. If you choose to update any of the object's user-configurable metadata (system or user-defined) during the copy, then you must explicitly specify all of the user-configurable metadata present on the source object in your request, even if you are changing only one of the metadata values. **Note** When copying an object by using the Amazon S3 console, you might receive the error message "Copied metadata can't be verified." The console uses headers to retrieve and set metadata for your object. If your network or browser configuration modifies your network requests, this behavior might cause unintended metadata (such as modified `Cache-Control` headers) to be written to your copied object. Amazon S3 can't verify this unintended metadata. To address this issue, check your network and browser configuration to make sure it doesn't modify headers, such as `Cache-Control`. For more information, see [The Shared Responsibility Model](https://docs.aws.amazon.com/whitepapers/latest/applying-security-practices-to-network-workload-for-csps/the-shared-responsibility-model.html). For more information about object metadata, see [Working with object metadata](UsingMetadata.md). **Copying archived and restored objects** If the source object is archived in S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive, you must first restore a temporary copy before you can copy the object to another bucket. For information about archiving objects, see [Working with archived objects](archived-objects.md). The **Copy** operation in the Amazon S3 console isn't supported for restored objects in the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes. To copy these restored objects, use the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. **Copying encrypted objects** Amazon S3 automatically encrypts all new objects that are copied to an S3 bucket. If you don't specify encryption information in your copy request, the encryption setting of the target object is set to the default encryption configuration of the destination bucket. By default, all buckets have a base level of encryption configuration that uses server-side encryption with Amazon S3 managed keys (SSE-S3). If the destination bucket has a default encryption configuration that uses server-side encryption with an AWS Key Management Service (AWS KMS) key (SSE-KMS), or a customer-provided encryption key (SSE-C), Amazon S3 uses the corresponding KMS key, or a customer-provided key to encrypt the target object copy. When copying an object, if you want to use a different type of encryption setting for the target object, you can request that Amazon S3 encrypt the target object with a KMS key, an Amazon S3 managed key, or a customer-provided key. If the encryption setting in your request is different from the default encryption configuration of the destination bucket, the encryption setting in your request takes precedence. If the source object for the copy is encrypted with SSE-C, you must provide the necessary encryption information in your request so that Amazon S3 can decrypt the object for copying. For more information, see [Protecting data with encryption](UsingEncryption.md). **Using checksums when copying objects** When copying objects, you can choose to use a different checksum algorithm for the object. Whether you choose to use the same algorithm or a new one, Amazon S3 calculates a new checksum value after the object is copied. Amazon S3 doesn't directly copy the value of the checksum. All copied objects without checksums and specified destination checksum algorithms automatically gain a `CRC-64NVME` checksum algorithm. For more information about how the checksum is calculated, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md). **Copying multiple objects in a single request** To copy more than one Amazon S3 object with a single request, you can also use S3 Batch Operations. You provide S3 Batch Operations with a list of objects to operate on. S3 Batch Operations calls the respective API operation to perform the specified operation. A single Batch Operations job can perform the specified operation on billions of objects containing exabytes of data. The S3 Batch Operations feature tracks progress, sends notifications, and stores a detailed completion report of all actions, providing a fully managed, auditable, serverless experience. You can use S3 Batch Operations through the Amazon S3 console, AWS CLI, AWS SDKs, or REST API. For more information, see [S3 Batch Operations basics](batch-ops.md#batch-ops-basics). **Copying objects to directory buckets** For information about copying an object to a directory bucket, see [Copying objects from or to a directory bucket](directory-buckets-objects-copy.md). For information about using the Amazon S3 Express One Zone storage class with directory buckets, see [S3 Express One Zone](directory-bucket-high-performance.md#s3-express-one-zone) and [Working with directory buckets](directory-buckets-overview.md). ## To copy an object To copy an object, use the following methods. ### Using the S3 console **Note** The restrictions and limitations when you copy an object with the console are as follows: You can copy an object if your object is less than 5 GB. If your object is greater than 5 GB, you must use the [AWS CLI](mpu-upload-object.md#UsingCLImpUpload) or [AWS SDKs](CopyingObjectsMPUapi.md) to copy an object. For a list of additional permissions required to copy objects, see [Required permissions for Amazon S3 API operations](using-with-s3-policy-actions.md). For example policies that grant this permission, see [Identity-based policy examples for Amazon S3](example-policies-s3.md). The `Copy` action applies to all objects within the specified folders (prefixes). Objects added to these folders while the action is in progress might be affected. Cross-Region copying of objects encrypted with SSE-KMS is not supported by the Amazon S3 console. To copy objects encrypted with SSE-KMS across Regions, use the AWS CLI, AWS SDK, or the Amazon S3 REST API. Objects encrypted with customer-provided encryption keys (SSE-C) cannot be copied by using the S3 console. To copy objects encrypted with SSE-C, use the AWS CLI, AWS SDK, or the Amazon S3 REST API. Copied objects will not retain the Object Lock settings from the original objects. If the bucket you are copying objects from uses the bucket owner enforced setting for S3 Object Ownership, object ACLs will not be copied to the specified destination. If you want to copy objects to a bucket that uses the bucket owner enforced setting for S3 Object Ownership, make sure that the source bucket also uses the bucket owner enforced setting, or remove any object ACL grants to other AWS accounts and groups. **To copy an object** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets** or **Directory buckets**. 1. In the list of buckets, choose the name of the bucket that contains the objects that you want to copy. 1. Select the check box to the left of the names of the objects that you want to copy. 1. On the **Actions** menu, choose **Copy** from the list of options that appears. 1. Select the destination type and destination account. To specify the destination path, choose **Browse S3**, navigate to the destination, and select the check box to the left of the destination. Choose **Choose destination** in the lower-right corner. Alternatively, enter the destination path. 1. If you do *not* have bucket versioning enabled, you will see a warning recommending you enable Bucket Versioning to help protect against unintentionally overwriting or deleting objects. If you want to keep all versions of objects in this bucket, select **Enable Bucket Versioning**. You can also view the default encryption and S3 Object Lock properties in **Destination details**. 1. Under **Additional copy settings**, choose whether you want to **Copy source settings**, **Don’t specify settings**, or **Specify settings**. **Copy source settings** is the default option. If you only want to copy the object without the source settings attributes, choose **Don’t specify settings**. Choose **Specify settings** to specify settings for storage class, ACLs, object tags, metadata, server-side encryption, and additional checksums. 1. Choose **Copy** in the bottom-right corner. Amazon S3 copies your objects to the destination. ### Using the AWS SDKs The examples in this section show how to copy objects up to 5 GB in a single operation. To copy objects larger than 5 GB, you must use a multipart upload. For more information, see [Copying an object using multipart upload](CopyingObjectsMPUapi.md). ------ #### [ Java ] For examples of how to copy objects with the AWS SDK for Java, see [Copy an object from one bucket to another](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_CopyObject_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] The following C\$1 example uses the high-level SDK for .NET to copy objects that are as large as 5 GB in a single operation. For objects that are larger than 5 GB, use the multipart upload copy example described in [Copying an object using multipart upload](CopyingObjectsMPUapi.md). This example makes a copy of an object that is a maximum of 5 GB. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Model; using System; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class CopyObjectTest { private const string sourceBucket = "*** provide the name of the bucket with source object ***"; private const string destinationBucket = "*** provide the name of the bucket to copy the object to ***"; private const string objectKey = "*** provide the name of object to copy ***"; private const string destObjectKey = "*** provide the destination object key name ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; public static void Main() { s3Client = new AmazonS3Client(bucketRegion); Console.WriteLine("Copying an object"); CopyingObjectAsync().Wait(); } private static async Task CopyingObjectAsync() { try { CopyObjectRequest request = new CopyObjectRequest { SourceBucket = sourceBucket, SourceKey = objectKey, DestinationBucket = destinationBucket, DestinationKey = destObjectKey }; CopyObjectResponse response = await s3Client.CopyObjectAsync(request); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } } } } ``` ------ #### [ PHP ] This topic guides you through using classes from version 3 of the AWS SDK for PHP to copy a single object and multiple objects within Amazon S3, from one bucket to another or within the same bucket. For more information about the AWS SDK for Ruby API, go to [AWS SDK for Ruby - Version 2](https://docs.aws.amazon.com/sdkforruby/api/index.html). The following PHP example illustrates the use of the `copyObject()` method to copy a single object within Amazon S3. It also demonstrates how to make multiple copies of an object by using a batch of calls to `CopyObject` with the `getcommand()` method. **Copying objects** | | | | --- |--- | | 1 | Create an instance of an Amazon S3 client by using the `Aws\S3\S3Client` class constructor. | | 2 | To make multiple copies of an object, you run a batch of calls to the Amazon S3 client [https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.AwsClientInterface.html#_getCommand](https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.AwsClientInterface.html#_getCommand) method, which is inherited from the [https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.CommandInterface.html](https://docs.aws.amazon.com/aws-sdk-php/v3/api/class-Aws.CommandInterface.html) class. You provide the `CopyObject` command as the first argument and an array containing the source bucket, source key name, target bucket, and target key name as the second argument. | ``` require 'vendor/autoload.php'; use Aws\CommandPool; use Aws\Exception\AwsException; use Aws\ResultInterface; use Aws\S3\S3Client; $sourceBucket = '*** Your Source Bucket Name ***'; $sourceKeyname = '*** Your Source Object Key ***'; $targetBucket = '*** Your Target Bucket Name ***'; $s3 = new S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); // Copy an object. $s3->copyObject([ 'Bucket' => $targetBucket, 'Key' => "$sourceKeyname-copy", 'CopySource' => "$sourceBucket/$sourceKeyname", ]); // Perform a batch of CopyObject operations. $batch = array(); for ($i = 1; $i <= 3; $i++) { $batch[] = $s3->getCommand('CopyObject', [ 'Bucket' => $targetBucket, 'Key' => "{targetKeyname}-$i", 'CopySource' => "$sourceBucket/$sourceKeyname", ]); } try { $results = CommandPool::batch($s3, $batch); foreach ($results as $result) { if ($result instanceof ResultInterface) { // Result handling here } if ($result instanceof AwsException) { // AwsException handling here } } } catch (Exception $e) { // General error handling here } ``` ------ #### [ Python ] ``` class ObjectWrapper: """Encapsulates S3 object actions.""" def __init__(self, s3_object): """ :param s3_object: A Boto3 Object resource. This is a high-level resource in Boto3 that wraps object actions in a class-like structure. """ self.object = s3_object self.key = self.object.key ``` ``` def copy(self, dest_object): """ Copies the object to another bucket. :param dest_object: The destination object initialized with a bucket and key. This is a Boto3 Object resource. """ try: dest_object.copy_from( CopySource={"Bucket": self.object.bucket_name, "Key": self.object.key} ) dest_object.wait_until_exists() logger.info( "Copied object from %s:%s to %s:%s.", self.object.bucket_name, self.object.key, dest_object.bucket_name, dest_object.key, ) except ClientError: logger.exception( "Couldn't copy object from %s/%s to %s/%s.", self.object.bucket_name, self.object.key, dest_object.bucket_name, dest_object.key, ) raise ``` ------ #### [ Ruby ] The following tasks guide you through using the Ruby classes to copy an object in Amazon S3 from one bucket to another or within the same bucket. **Copying objects** | | | | --- |--- | | 1 | Use the Amazon S3 modularized gem for version 3 of the AWS SDK for Ruby, require `aws-sdk-s3`, and provide your AWS credentials. For more information about how to provide your credentials, see [Making requests using AWS account or IAM user credentials](https://docs.aws.amazon.com/AmazonS3/latest/API/AuthUsingAcctOrUserCredentials.html) in the *Amazon S3 API Reference*. | | 2 | Provide the request information, such as the source bucket name, source key name, destination bucket name, and destination key. | The following Ruby code example demonstrates the preceding tasks by using the `#copy_object` method to copy an object from one bucket to another. ``` require 'aws-sdk-s3' # Wraps Amazon S3 object actions. class ObjectCopyWrapper attr_reader :source_object # @param source_object [Aws::S3::Object] An existing Amazon S3 object. This is used as the source object for # copy actions. def initialize(source_object) @source_object = source_object end # Copy the source object to the specified target bucket and rename it with the target key. # # @param target_bucket [Aws::S3::Bucket] An existing Amazon S3 bucket where the object is copied. # @param target_object_key [String] The key to give the copy of the object. # @return [Aws::S3::Object, nil] The copied object when successful; otherwise, nil. def copy_object(target_bucket, target_object_key) @source_object.copy_to(bucket: target_bucket.name, key: target_object_key) target_bucket.object(target_object_key) rescue Aws::Errors::ServiceError => e puts "Couldn't copy #{@source_object.key} to #{target_object_key}. Here's why: #{e.message}" end end # Example usage: def run_demo source_bucket_name = "amzn-s3-demo-bucket1" source_key = "my-source-file.txt" target_bucket_name = "amzn-s3-demo-bucket2" target_key = "my-target-file.txt" source_bucket = Aws::S3::Bucket.new(source_bucket_name) wrapper = ObjectCopyWrapper.new(source_bucket.object(source_key)) target_bucket = Aws::S3::Bucket.new(target_bucket_name) target_object = wrapper.copy_object(target_bucket, target_key) return unless target_object puts "Copied #{source_key} from #{source_bucket_name} to #{target_object.bucket_name}:#{target_object.key}." end run_demo if $PROGRAM_NAME == __FILE__ ``` ------ ### Using the REST API This example describes how to copy an object by using the Amazon S3 REST API. For more information about the REST API, see [https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html). This example copies the `flotsam` object from the `amzn-s3-demo-bucket1` bucket to the `jetsam` object of the `amzn-s3-demo-bucket2` bucket, preserving its metadata. ``` 1. PUT /jetsam HTTP/1.1 2. Host: amzn-s3-demo-bucket2.s3.amazonaws.com 3. x-amz-copy-source: /amzn-s3-demo-bucket1/flotsam 4. Authorization: AWS AKIAIOSFODNN7EXAMPLE:ENoSbxYByFA0UGLZUqJN5EUnLDg= 5. Date: Wed, 20 Feb 2008 22:12:21 +0000 ``` The signature was generated from the following information. ``` 1. PUT\r\n 2. \r\n 3. \r\n 4. Wed, 20 Feb 2008 22:12:21 +0000\r\n 5. 6. x-amz-copy-source:/amzn-s3-demo-bucket1/flotsam\r\n 7. /amzn-s3-demo-bucket2/jetsam ``` Amazon S3 returns the following response that specifies the ETag of the object and when it was last modified. ``` 1. HTTP/1.1 200 OK 2. x-amz-id-2: Vyaxt7qEbzv34BnSu5hctyyNSlHTYZFMWK4FtzO+iX8JQNyaLdTshL0KxatbaOZt 3. x-amz-request-id: 6B13C3C5B34AF333 4. Date: Wed, 20 Feb 2008 22:13:01 +0000 5. 6. Content-Type: application/xml 7. Transfer-Encoding: chunked 8. Connection: close 9. Server: AmazonS3 10. 11. 12. 13. 2008-02-20T22:13:01 14. "7e9c608af58950deeb370c98608ed097" 15. ``` ### Using the AWS CLI You can also use the AWS Command Line Interface (AWS CLI) to copy an S3 object. For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/copy-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/copy-object.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ## To move an object To move an object, use the following methods. ### Using the S3 console **Note** You can move an object if your object is less than 5 GB. If your object is greater than 5 GB, you must use the [AWS CLI](mpu-upload-object.md#UsingCLImpUpload) or [AWS SDKs](CopyingObjectsMPUapi.md) to move an object. For a list of additional permissions required to move objects, see [Required permissions for Amazon S3 API operations](using-with-s3-policy-actions.md). For example policies that grant this permission, see [Identity-based policy examples for Amazon S3](example-policies-s3.md). Objects encrypted with customer-provided encryption keys (SSE-C) can't be moved by using the Amazon S3 console. To move objects encrypted with SSE-C, use the AWS CLI, AWS SDKs, or the Amazon S3 REST API. When moving folders, wait for the **Move** operation to finish before making additional changes in the folders. You can't use S3 access point aliases as the source or destination for **Move** operations in the Amazon S3 console. **To move an object** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. Navigate to the Amazon S3 bucket or folder that contains the objects that you want to move. 1. Select the check box for the objects that you want to move. 1. On the **Actions** menu, choose **Move**. 1. To specify the destination path, choose **Browse S3**, navigate to the destination, and select the destination check box. Choose **Choose destination**. Alternatively, enter the destination path. 1. If you do *not* have bucket versioning enabled, you will see a warning recommending you enable Bucket Versioning to help protect against unintentionally overwriting or deleting objects. If you want to keep all versions of objects in this bucket, select **Enable Bucket Versioning**. You can also view the default encryption and Object Lock properties in **Destination details**. 1. Under **Additional copy settings**, choose whether you want to **Copy source settings**, **Don’t specify settings**, or **Specify settings**. **Copy source settings** is the default option. If you only want to copy the object without the source settings attributes, choose **Don’t specify settings**. Choose **Specify settings** to specify settings for storage class, ACLs, object tags, metadata, server-side encryption, and additional checksums. 1. Choose **Move** in the bottom-right corner. Amazon S3 moves your objects to the destination. ### Using the AWS CLI You can also use the AWS Command Line Interface (AWS CLI) to move an S3 object. For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/mv.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/mv.html) in the *AWS CLI Command Reference*. For information about the AWS CLI, see [What is the AWS Command Line Interface?](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) in the *AWS Command Line Interface User Guide*. ## To rename an object To rename an object, use the following procedure. **Note** You can rename an object if your object is less than 5 GB. To rename objects greater than 5 GB, you must use the [AWS CLI](mpu-upload-object.md#UsingCLImpUpload) or [AWS SDKs](CopyingObjectsMPUapi.md) to copy your object with a new name and then delete the original object. For a list of additional permissions required to copy objects, see [Required permissions for Amazon S3 API operations](using-with-s3-policy-actions.md). For example policies that grant this permission, see [Identity-based policy examples for Amazon S3](example-policies-s3.md). Renaming an object creates a copy of the object with a new last-modified date, and then adds a delete marker to the original object. Bucket settings for default encryption are automatically applied to any specified object that's unencrypted. You can't use the Amazon S3 console to rename objects with customer-provided encryption keys (SSE-C). To rename objects encrypted with SSE-C, use the AWS CLI, AWS SDKs, or the Amazon S3 REST API to copy those objects with new names. If this bucket uses the bucket owner enforced setting for S3 Object Ownership, object access control lists (ACLs) won't be copied. **To rename an object** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the navigation pane, choose **Buckets**, and then choose the **General purpose buckets** tab. Navigate to the Amazon S3 bucket or folder that contains the object that you want to rename. 1. Select the check box for the object that you want to rename. 1. On the **Actions** menu, choose **Rename object**. 1. In the **New object name** box, enter the new name for the object. 1. Under **Additional copy settings**, choose whether you want to **Copy source settings**, **Don’t specify settings**, or **Specify settings**. **Copy source settings** is the default option. If you only want to copy the object without the source settings attributes, choose **Don’t specify settings**. Choose **Specify settings** to specify settings for storage class, ACLs, object tags, metadata, server-side encryption, and additional checksums. 1. Choose **Save changes**. Amazon S3 renames your object. # Downloading objects This section explains how to download objects from an Amazon S3 bucket. With Amazon S3, you can store objects in one or more buckets, and each single object can be up to 50 TB in size. Any Amazon S3 object that is not archived is accessible in real time. Archived objects, however, must be restored before they can be downloaded. For information about downloading archived objects, see [Downloading archived objects](#download-archived-objects). You can download a single object by using the Amazon S3 console, AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. To download an object up to 5 TB from S3 without writing any code or running any commands, use the S3 console. For more information, see [Downloading an object](#download-an-object). To download objects larger than 5 TB, use concurrent `GetObject` requests with either `Range` HTTP header to read specific byte ranges or `partNumber` to download specific part of an object. Single GET requests are limited to 5 TB, and you will receive a `405 - Method Not Allowed` error for GET requests beyond 5 TB. For large object downloads, use the S3 Transfer Manager in the Java v1/v2, Python, or AWS CLI SDKs. For the best performance, use the latest AWS Common Runtime (CRT) with these SDKs, which has been optimized for better resource utilization. CRT automatically scales the size of individual GETs to optimize throughput. You can improve overall transfer throughput by allocating more memory using memory limit parameters such as `maxNativeMemoryLimitInBytes` for Java SDK. You can opt out of this behavior by setting an explicit part size using request parameters such as `multipart_chunksize` for AWS CLI and `minimumPartSizeInBytes` for Java SDK in your download request. To download multiple objects, use AWS CloudShell, the AWS CLI, or the AWS SDKs. For more information, see [Downloading multiple objects](#download-multiple-objects). If you need to download part of an object, you use extra parameters with the AWS CLI or REST API to specify only the bytes that you want to download. For more information, see [Downloading part of an object](#download-objects-parts). If you need to download an object that you don't own, ask the object owner to generate a presigned URL that allows you to download the object. For more information, see [Downloading an object from another AWS account](#download-objects-from-another-account). When you download objects outside of the AWS network, data-transfer fees apply. Data transfer within the AWS network is free within the same AWS Region, but you will be charged for any `GET` requests. For more information about data-transfer costs and data-retrieval charges, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/). **Topics** + [ ## Downloading an object ](#download-an-object) + [ ## Downloading multiple objects ](#download-multiple-objects) + [ ## Downloading part of an object ](#download-objects-parts) + [ ## Downloading an object from another AWS account ](#download-objects-from-another-account) + [ ## Downloading archived objects ](#download-archived-objects) + [ ## Downloading objects based on metadata ](#download-objects-based-on-metadata) + [ ## Troubleshooting downloading objects ](#download-objects-troubleshooting) ## Downloading an object You can download an object by using the Amazon S3 console, AWS CLI, AWS SDKs, or REST API. ### Using the S3 console This section explains how to use the Amazon S3 console to download an object from an S3 bucket. **Note** You can download only one object at a time. If you use the Amazon S3 console to download an object whose key name ends with a period (`.`), the period is removed from the key name of the downloaded object. To retain the period at the end of the name of the downloaded object, you must use the AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. **To download an object from an S3 bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets** or **Directory buckets**. 1. In the buckets list, choose the name of the bucket that you want to download an object from. 1. You can download an object from an S3 bucket in any of the following ways: + Select the check box next to the object, and choose **Download**. If you want to download the object to a specific folder, on the **Actions** menu, choose **Download as**. + If you want to download a specific version of the object, turn on **Show versions** (located next to the search box). Select the check box next to the version of the object that you want, and choose **Download**. If you want to download the object to a specific folder, on the **Actions** menu, choose **Download as**. ### Using the AWS CLI The following `get-object` example command shows how you can use the AWS CLI to download an object from Amazon S3. This command gets the object `folder/my_image` from the bucket `amzn-s3-demo-bucket1`. You must include an `outfile`, which is a file name for the downloaded object, such as `my_downloaded_image.jpg`. ``` aws s3api get-object --bucket amzn-s3-demo-bucket1 --key folder/my_image my_downloaded_image.jpg ``` For more information and examples, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/get-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/get-object.html) in the *AWS CLI Command Reference*. ### Using the AWS SDKs For examples of how to download an object with the AWS SDKs, see [Code examples](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_GetObject_section.html) in the *Amazon S3 API Reference*. For general information about using different AWS SDKs, see [Developing with Amazon S3 using the AWS SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/sdk-general-information-section.html) in the *Amazon S3 API Reference*. ### Using the REST API You can use the REST API to retrieve objects from Amazon S3. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) in the *Amazon Simple Storage Service API Reference*. ## Downloading multiple objects You can download multiple objects by using AWS CloudShell, the AWS CLI, or the AWS SDKs. ### Using AWS CloudShell in the AWS Management Console AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information about AWS CloudShell, see [What is CloudShell?](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html) in the *AWS CloudShell User Guide*. **Important** With AWS CloudShell, your home directory has storage up to 1GB per AWS Region. Therefore you cannot sync buckets with objects totaling over this amount. For more limitations, see [Service quotas and restrictions](https://docs.aws.amazon.com/cloudshell/latest/userguide/limits.html) in the *AWS CloudShell User Guide*. **To download objects by using AWS CloudShell** 1. Sign in to the AWS Management Console and open the CloudShell console at [https://console.aws.amazon.com/cloudshell/](https://console.aws.amazon.com/cloudshell/). 1. Run the following command to sync objects in your bucket to CloudShell. The following command syncs objects from the bucket named `amzn-s3-demo-bucket1` and creates a folder named `temp` in CloudShell. CloudShell syncs your objects to this folder. To use this command, replace the `user input placeholders` with your own information. ``` aws s3 sync s3://amzn-s3-demo-bucket1 ./temp ``` **Note** The `sync` command is not compatible with directory buckets. To perform pattern matching to either exclude or include particular objects, you can use the `--exclude "value"` and `--include "value"` parameters with the `sync` command. 1. Run the following command to zip your objects in the folder named `temp` to a file named `temp.zip`. ``` zip temp.zip -r temp/ ``` 1. Choose **Actions**, and then choose **Download file**. 1. Enter the file name `temp.zip` and then choose **Download**. 1. (Optional) Delete the `temp.zip` file and the objects that are synced to the `temp` folder in CloudShell. With AWS CloudShell, you have persistent storage of up to 1 GB for each AWS Region. You can use the following example command to delete your `.zip` file and your folder. To use this example command, replace the `user input placeholders` with your own information. ``` rm temp.zip && rm -rf temp/ ``` ### Using the AWS CLI The following example shows how you can use the AWS CLI to download all of the files or objects under the specified directory or prefix. This command copies all objects from the bucket `amzn-s3-demo-bucket1` to your current directory. To use this example command, use your bucket name in place of `amzn-s3-demo-bucket1`. ``` aws s3 cp s3://amzn-s3-demo-bucket1 . --recursive ``` The following command downloads all of the objects under the prefix `logs` in the bucket `amzn-s3-demo-bucket1` to your current directory. It also uses the `--exclude` and `--include` parameters to copy only objects with the suffix `.log`. To use this example command, replace the `user input placeholders` with your own information. ``` aws s3 cp s3://amzn-s3-demo-bucket1/logs/ . --recursive --exclude "*" --include "*.log" ``` For more information and examples, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html) in the *AWS CLI Command Reference*. ### Using the AWS SDKs For examples of how to download all objects in an Amazon S3 bucket with the AWS SDKs, see [Code examples](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_DownloadBucketToDirectory_section.html) in the *Amazon S3 API Reference*. For general information about using different AWS SDKs, see [Developing with Amazon S3 using the AWS SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/sdk-general-information-section.html) in the *Amazon S3 API Reference*. ## Downloading part of an object You can download part of an object by using the AWS CLI or REST API. To do so, you use additional parameters to specify which part of an object that you want to download. ### Using the AWS CLI The following example command performs a `GET` request for a range of bytes in the object named `folder/my_data` in the bucket named `amzn-s3-demo-bucket1`. In the request, the byte range must be prefixed with `bytes=`. The partial object is downloaded to the output file named `my_data_range`. To use this example command, replace the `user input placeholders` with your own information. ``` aws s3api get-object --bucket amzn-s3-demo-bucket1 --key folder/my_data --range bytes=0-500 my_data_range ``` For more information and examples, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/get-object.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/get-object.html) in the *AWS CLI Command Reference*. For more information about the HTTP `Range` header, see [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) on the RFC Editor website. **Note** Amazon S3 doesn't support retrieving multiple ranges of data in a single `GET` request. ### Using the REST API You can use the `partNumber` and `Range` parameters in the REST API to retrieve object parts from Amazon S3. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) in the *Amazon Simple Storage Service API Reference*. ## Downloading an object from another AWS account You can use a presigned URL to grant others time-limited access to your objects without updating your bucket policy. The presigned URL can be entered in a browser or used by a program to download an object. The credentials used by the URL are those of the AWS user who generated the URL. After the URL is created, anyone with the presigned URL can download the corresponding object until the URL expires. ### Using a presigned URL in the S3 console You can use the Amazon S3 console to generate a presigned URL for sharing an object in a general purpose bucket by following these steps. When using the console, the maximum expiration time for a presigned URL is 12 hours from the time of creation. **To generate a presigned URL by using the Amazon S3 console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the buckets list, choose the name of the bucket that contains the object that you want a presigned URL for. 1. In the **Objects** list, select the object that you want to create a presigned URL for. 1. On the **Object actions** menu, choose **Share with a presigned URL**. 1. Specify how long you want the presigned URL to be valid. 1. Choose **Create presigned URL**. 1. When a confirmation message appears, the URL is automatically copied to your clipboard. You will see a button to copy the presigned URL if you need to copy it again. 1. To download the object, paste the URL into any browser, and the object will attempt to download. For more information about presigned URLs and other methods for creating them, see [Download and upload objects with presigned URLs](using-presigned-url.md). ## Downloading archived objects To reduce your storage costs for infrequently accessed objects, you can *archive* those objects. When you archive an object, it is moved into low-cost storage, which means that you can't access it in real time. To download an archived object, you must first restore it. You can restore archived objects in minutes or hours, depending on the storage class. You can restore an archived object by using the Amazon S3 console, S3 Batch Operations, the Amazon S3 REST API, the AWS SDKs, and the AWS Command Line Interface (AWS CLI). For instructions, see [Restoring an archived object](restoring-objects.md). After you restore the archived object, you can download it. ## Downloading objects based on metadata You can add preconditions to download an object based on it's metadata using a conditional read request. You can return an object based on it's Entity tag (ETag) or last modified date. This can limit an S3 operation to objects updated since a specified date or only return a specific object version. You can use conditional writes for [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) or [https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) requests. For more information about conditional requests see, [Add preconditions to S3 operations with conditional requests](conditional-requests.md). ## Troubleshooting downloading objects Insufficient permissions or incorrect bucket or AWS Identity and Access Management (IAM) user policies can cause errors when you're trying to download objects from Amazon S3. These problems can often cause Access Denied (403 Forbidden) errors, where Amazon S3 is unable to allow access to a resource. For common causes of Access Denied (403 Forbidden) errors, see [Troubleshoot access denied (403 Forbidden) errors in Amazon S3](troubleshoot-403-errors.md). If you're getting a 404 NoSuchKey error when trying to access an object, see [How can I troubleshoot the 404 NoSuchKey error from Amazon S3?](https://repost.aws/knowledge-center/404-error-nosuchkey-s3) in the AWS re:Post Knowledge Center. # Checking object integrity in Amazon S3 Amazon S3 provides a range of data protection capabilities throughout an object's storage lifecycle. With Amazon S3, you can use checksum values to verify the integrity of the data that you upload or download. In addition, you can request that another checksum value be calculated for any object that you store in S3. When uploading, copying, or managing your data, you can choose from several supported checksum algorithms: + CRC-64/NVME (`CRC64NVME`) **Note** The `CRC64NVME` checksum algorithm is the default checksum algorithm used for checksum calculations. + CRC-32 (`CRC32`) + CRC-32C (`CRC32C`) + SHA-1 (`SHA1`) + SHA-256 (`SHA256`) + MD5 (`MD5`) **Note** For multipart uploads, the Compute checksum operation provides full object checksum values using `MD5`, which isn’t possible during uploads. For single part uploads, the `content-MD5 header` is only available using the S3 ETag for objects and must use SSE-S3 encryption. When you upload an object to S3, you can specify the usage of any of these checksum algorithms. For uploads, all AWS-owned clients calculate a checksum of the object and send it with the upload request. S3 then independently calculates a checksum value of the object on the server-side, and validates it with the provided value before storing the object and checksum value. You can also provide precalculated values for these checksum algorithms when you a perform a single part upload or multipart upload (using the full object checksum type for multipart uploads). To use precalculated values with multiple objects, use the AWS CLI or AWS SDKs. Alternatively, if you want to verify datasets in S3 without the need to restore or download data, you can use the **Compute checksum** operation with S3 Batch Operations. The **Compute checksum** operation allows you to efficiently verify billions of objects in one job request. When the **Compute checksum** operation is performed, S3 calculates the checksum values for a list of objects, at rest. At the end of the job request, you’ll receive an automatically generated integrity report (also known as a completion report) that you can use to confirm that your data set remains intact. **Topics** + [ # Checking object integrity for data uploads in Amazon S3 ](checking-object-integrity-upload.md) + [ # Checking object integrity for data at rest in Amazon S3 ](checking-object-integrity-at-rest.md) # Checking object integrity for data uploads in Amazon S3 Amazon S3 uses checksum values to verify data integrity during upload and download operations. When you upload data, AWS SDK and the AWS Management Console use your chosen checksum algorithm to compute a checksum value before data transmission. S3 then independently calculates a checksum of your data and validates it against the provided checksum value. Objects are accepted only after confirming data integrity was maintained during transit. S3 stores both the checksum value as object metadata and the object itself. To verify object integrity, you can request the checksum value during downloads. This validation works consistently across encryption modes, object sizes, storage classes, and both single part and multipart uploads. To change the checksum algorithm for an upload, you can copy an individual object or use batch copy for multiple objects. For single part uploads, you can provide checksum values as headers. You can either provide a pre-calculated value or let the AWS SDK calculate one during upload. If the checksum value calculated by S3 matches your provided value, the request is accepted. If the values don't match, the request is rejected. For multipart uploads, AWS SDKs can automatically create trailing checksums for chunked uploads. When you use a trailing checksum, Amazon S3 generates checksum values for every part using your specified algorithm and appends the checksum value to the end of the chunked upload request. S3 performs the verification and upload in a single pass, improving efficiency. For more information, see see [Using trailing checksums](#trailing-checksums). ## Using supported checksum algorithms With Amazon S3, you can choose a checksum algorithm to calculate the checksum values during uploads. The specified checksum algorithm is then stored with your object and can be used to validate data integrity during downloads. You can choose one of the following Secure Hash Algorithms (SHA) or Cyclic Redundancy Check (CRC) checksum algorithms to calculate the checksum value: + CRC-64/NVME (`CRC64NVME`) + CRC-32 (`CRC32`) + CRC-32C (`CRC32C`) + SHA-1 (`SHA1`) + SHA-256 (`SHA256`) + MD5 (`MD5`) **Note** The `content-MD5` header is only available using the S3 ETag for objects uploaded in a single part upload (`PUT` operation) that uses the SSE-S3 encryption. Additionally, you can provide a checksum with each request using the Content-MD5 header. When you [upload an object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html), you specify the algorithm that you want to use: + **When you use the AWS Management Console**, choose the checksum algorithm that you want to use. You can optionally specify the checksum value of the object. When Amazon S3 receives the object, it calculates the checksum by using the algorithm that you specified. If the two checksum values don't match, Amazon S3 generates an error. + **When you use an SDK**, be aware of the following: + Set the `ChecksumAlgorithm` parameter to the algorithm that you want Amazon S3 to use. If you already have a precalculated checksum, you pass the checksum value to the AWS SDK, and the SDK includes the value in the request. If you don’t pass a checksum value or don’t specify a checksum algorithm, the SDK automatically calculates a checksum value for you and includes it with the request to provide integrity protections. If the individual checksum value doesn't match the set value of the checksum algorithm, Amazon S3 fails the request with a `BadDigest` error. + If you’re using an upgraded AWS SDK, the SDK chooses a checksum algorithm for you. However, you can override this checksum algorithm. + If you don’t specify a checksum algorithm and the SDK also doesn’t calculate a checksum for you, then S3 automatically chooses the CRC-64/NVME (`CRC64NVME`) checksum algorithm. + **When you use the REST API**, don't use the `x-amz-sdk-checksum-algorithm` parameter. Instead, use one of the algorithm-specific headers (for example, `x-amz-checksum-crc32`). To apply any of these checksum values to objects that are already uploaded to Amazon S3, you can copy the object and specify whether you want to use the existing checksum algorithm or a new one. If you don’t specify an algorithm, S3 uses the existing algorithm. If the source object doesn’t have a specified checksum algorithm or checksum value, Amazon S3 uses the CRC-64/NVME algorithm to calculate the checksum value for the destination object. You can also specify a checksum algorithm when copying objects using [S3 Batch Operations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops.html). **Important** If you use a multipart upload with **Checksums** for composite (or part-level) checksums, the multipart upload part numbers must be consecutive and begin with 1. If you try to complete a multipart upload request with nonconsecutive part numbers, Amazon S3 generates an `HTTP 500 Internal Server` error. ## Full object and composite checksum types In Amazon S3, there are two types of supported checksums: + **Full object checksums:** A full object checksum is calculated based on all of the content of a multipart upload, covering all data from the first byte of the first part to the last byte of the last part. Be aware that if you're using the AWS Management Console to upload objects that are smaller than 16 MB, only the full object checksum type is supported. **Note** All PUT requests require a full object checksum type. You must specify a full object checksum type if you're uploading your object via PUT request. + **Composite checksums:** A composite checksum is calculated based on the individual checksums of each part in a multipart upload. Instead of computing a checksum based on all of the data content, this approach aggregates the part-level checksums (from the first part to the last) to produce a single, combined checksum for the complete object. If you're using a multipart upload to upload your object, then you must specify the composite checksum type. **Note** When an object is uploaded as a multipart upload, the entity tag (ETag) for the object is not an MD5 digest of the entire object. Instead, Amazon S3 calculates the MD5 digest of each individual part as it is uploaded. The MD5 digests are used to determine the ETag for the final object. Amazon S3 concatenates the bytes for the MD5 digests together and then calculates the MD5 digest of these concatenated values. During the final ETag creation step, Amazon S3 adds a dash with the total number of parts to the end. Amazon S3 supports the following full object and composite checksum algorithm types: + CRC-64/NVME (`CRC64NVME`): Supports the full object checksum type only. + CRC-32 (`CRC32`): Supports both full object and composite checksum types. + CRC-32C (`CRC32C`): Supports both full object and composite checksum types. + SHA-1 (`SHA1`): Supports both full object and composite checksum types. + SHA-256 (`SHA256`): Supports both full object and composite checksum types. + MD5 (`MD5`): Supports both full object and composite checksum types. ### Single part uploads Checksums of objects that are uploaded in a single part (using [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject)) are treated as full object checksums. When you upload an object in the Amazon S3 console, you can choose the checksum algorithm that you want S3 to use and also (optionally) provide a precomputed value. Amazon S3 then validates the precomputed checksum value before storing the object and its checksum value. You can verify an object's data integrity when you request the checksum value during object downloads. ### Multipart uploads When you upload the object in multiple parts using the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_MultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_MultipartUpload.html) API, you can specify the checksum algorithm that you want Amazon S3 to use and the checksum type (full object or composite). The following table indicates which checksum algorithm type is supported for each checksum algorithm in a multipart upload: | Checksum algorithm | Full object | Composite | | --- | --- | --- | | CRC-64/NVME (CRC64NVME) | Yes | No | | CRC-32 (CRC32) | Yes | Yes | | CRC-32C (CRC32C) | Yes | Yes | | SHA-1 (SHA1) | No | Yes | | SHA-256 (SHA256) | No | Yes | ## Using full object checksums for multipart upload When creating or performing a multipart upload, you can use full object checksums for validation on upload. This means that you can provide the checksum algorithm for the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_MultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_MultipartUpload.html) API, simplifying your integrity validation tooling because you no longer need to track part boundaries for uploaded objects. You can provide the checksum of the whole object in the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) request, along with the object size. When you provide a full object checksum during a multipart upload, the AWS SDK passes the checksum to Amazon S3, and S3 validates the object integrity server-side, comparing it to the received value. Then, Amazon S3 stores the object if the values match. If the two values don’t match, S3 fails the request with a `BadDigest` error. The checksum of your object is also stored in object metadata that you use later to validate an object's data integrity. For full object checksums, you can use CRC-64/NVME (`CRC64NVME`), CRC-32 (`CRC32`), or CRC-32C (`CRC32C`) checksum algorithms in S3. Full object checksums in multipart uploads are only available for CRC-based checksums because they can linearize into a full object checksum. This linearization allows Amazon S3 to parallelize your requests for improved performance. In particular, S3 can compute the checksum of the whole object from the part-level checksums. This type of validation isn’t available for other algorithms, such as SHA and MD5. Because S3 has default integrity protections, if objects are uploaded without a checksum, S3 automatically attaches the recommended full object CRC-64/NVME (`CRC64NVME`) checksum algorithm to the object. **Note** To initiate the multipart upload, you can specify the checksum algorithm and the full object checksum type. After you specify the checksum algorithm and the full object checksum type, you can provide the full object checksum value for the multipart upload. ## Using part-level checksums for multipart upload When objects are uploaded to Amazon S3, they can be uploaded as a single object or uploaded in parts with the multipart upload process. You can choose a **Checksum** type for your multipart upload. For multipart upload part-level checksums (or composite checksums), Amazon S3 calculates the checksum for each individual part by using the specified checksum algorithm. You can use [https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) to provide the checksum values for each part. If the object that you try to upload in the Amazon S3 console is set to use the CRC-64/NVME (`CRC64NVME`) checksum algorithm and exceeds 16 MB, it is automatically designated as a full object checksum. Amazon S3 then uses the stored part-level checksum values to confirm that each part is uploaded correctly. When each part’s checksum (for the whole object) is provided, S3 uses the stored checksum values of each part to calculate the full object checksum internally, comparing it with the provided checksum value. This minimizes compute costs since S3 can compute a checksum of the whole object using the checksum of the parts. For more information about multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](mpuoverview.md) and [Using full object checksums for multipart upload](#Full-object-checksums). When the object is completely uploaded, you can use the final calculated checksum to verify the data integrity of the object. When uploading a part of the multipart upload, be aware of the following: + To retrieve information about the object, including how many parts make up the entire object, you can use the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html) operation. With additional checksums, you can also recover information for each individual part that includes the part's checksum value. + For completed uploads, you can get an individual part's checksum by using the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) or [https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) operations and specifying a part number or byte range that aligns with a single part. If you want to retrieve the checksum values for individual parts of multipart uploads that are still in progress, you can use [https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html). + Because of how Amazon S3 calculates the checksum for multipart objects, the checksum value for the object might change if you copy it. If you're using an SDK or the REST API and you call [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html), Amazon S3 copies any object up to the size limitations of the `CopyObject` API operation. Amazon S3 does this copy as a single action, regardless of whether the object was uploaded in a single request or as part of a multipart upload. With a copy command, the checksum of the object is a direct checksum of the full object. If the object was originally uploaded using a multipart upload, the checksum value changes even though the data doesn't. + Objects that are larger than the size limitations of the `CopyObject` API operation must use [multipart upload copy commands](https://docs.aws.amazon.com/AmazonS3/latest/userguide/CopyingObjectsMPUapi.html). + When you perform some operations using the AWS Management Console, Amazon S3 uses a multipart upload if the object is greater than 16 MB in size. ## Checksum methods After uploading objects, you can get the checksum value and compare it to a precomputed or previously stored checksum value of the same checksum algorithm type. The following examples show you which checksum calculation methods you can use to verify data integrity. ### Using the S3 console To learn more about using the console and specifying checksum algorithms to use when uploading objects, see [Uploading objects](upload-objects.md) and [Tutorial: Checking the integrity of data in Amazon S3 with additional checksums](https://aws.amazon.com/getting-started/hands-on/amazon-s3-with-additional-checksums/?ref=docs_gateway/amazons3/checking-object-integrity.html). ### Using the AWS SDKs The following example shows how you can use the AWS SDKs to upload a large file with multipart upload, download a large file, and validate a multipart upload file, all by using SHA-256 for file validation. ------ #### [ Java ] **Example: Uploading, downloading, and verifying a large file with SHA-256** For instructions on creating and testing a working sample, see [Getting Started](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/getting-started.html) in the AWS SDK for Java Developer Guide. ``` import software.amazon.awssdk.auth.credentials.AwsCredentials; import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; import software.amazon.awssdk.core.ResponseInputStream; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest; import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm; import software.amazon.awssdk.services.s3.model.ChecksumMode; import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest; import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse; import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload; import software.amazon.awssdk.services.s3.model.CompletedPart; import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest; import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse; import software.amazon.awssdk.services.s3.model.GetObjectAttributesRequest; import software.amazon.awssdk.services.s3.model.GetObjectAttributesResponse; import software.amazon.awssdk.services.s3.model.GetObjectRequest; import software.amazon.awssdk.services.s3.model.GetObjectResponse; import software.amazon.awssdk.services.s3.model.GetObjectTaggingRequest; import software.amazon.awssdk.services.s3.model.ObjectAttributes; import software.amazon.awssdk.services.s3.model.PutObjectTaggingRequest; import software.amazon.awssdk.services.s3.model.Tag; import software.amazon.awssdk.services.s3.model.Tagging; import software.amazon.awssdk.services.s3.model.UploadPartRequest; import software.amazon.awssdk.services.s3.model.UploadPartResponse; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; import java.nio.ByteBuffer; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; import java.util.ArrayList; import java.util.Base64; import java.util.List; public class LargeObjectValidation { private static String FILE_NAME = "sample.file"; private static String BUCKET = "sample-bucket"; //Optional, if you want a method of storing the full multipart object checksum in S3. private static String CHECKSUM_TAG_KEYNAME = "fullObjectChecksum"; //If you have existing full-object checksums that you need to validate against, you can do the full object validation on a sequential upload. private static String SHA256_FILE_BYTES = "htCM5g7ZNdoSw8bN/mkgiAhXt5MFoVowVg+LE9aIQmI="; //Example Chunk Size - this must be greater than or equal to 5MB. private static int CHUNK_SIZE = 5 * 1024 * 1024; public static void main(String[] args) { S3Client s3Client = S3Client.builder() .region(Region.US_EAST_1) .credentialsProvider(new AwsCredentialsProvider() { @Override public AwsCredentials resolveCredentials() { return new AwsCredentials() { @Override public String accessKeyId() { return Constants.ACCESS_KEY; } @Override public String secretAccessKey() { return Constants.SECRET; } }; } }) .build(); uploadLargeFileBracketedByChecksum(s3Client); downloadLargeFileBracketedByChecksum(s3Client); validateExistingFileAgainstS3Checksum(s3Client); } public static void uploadLargeFileBracketedByChecksum(S3Client s3Client) { System.out.println("Starting uploading file validation"); File file = new File(FILE_NAME); try (InputStream in = new FileInputStream(file)) { MessageDigest sha256 = MessageDigest.getInstance("SHA-256"); CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder() .bucket(BUCKET) .key(FILE_NAME) .checksumAlgorithm(ChecksumAlgorithm.SHA256) .build(); CreateMultipartUploadResponse createdUpload = s3Client.createMultipartUpload(createMultipartUploadRequest); List completedParts = new ArrayList(); int partNumber = 1; byte[] buffer = new byte[CHUNK_SIZE]; int read = in.read(buffer); while (read != -1) { UploadPartRequest uploadPartRequest = UploadPartRequest.builder() .partNumber(partNumber).uploadId(createdUpload.uploadId()).key(FILE_NAME).bucket(BUCKET).checksumAlgorithm(ChecksumAlgorithm.SHA256).build(); UploadPartResponse uploadedPart = s3Client.uploadPart(uploadPartRequest, RequestBody.fromByteBuffer(ByteBuffer.wrap(buffer, 0, read))); CompletedPart part = CompletedPart.builder().partNumber(partNumber).checksumSHA256(uploadedPart.checksumSHA256()).eTag(uploadedPart.eTag()).build(); completedParts.add(part); sha256.update(buffer, 0, read); read = in.read(buffer); partNumber++; } String fullObjectChecksum = Base64.getEncoder().encodeToString(sha256.digest()); if (!fullObjectChecksum.equals(SHA256_FILE_BYTES)) { //Because the SHA256 is uploaded after the part is uploaded; the upload is bracketed and the full object can be fully validated. s3Client.abortMultipartUpload(AbortMultipartUploadRequest.builder().bucket(BUCKET).key(FILE_NAME).uploadId(createdUpload.uploadId()).build()); throw new IOException("Byte mismatch between stored checksum and upload, do not proceed with upload and cleanup"); } CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder().parts(completedParts).build(); CompleteMultipartUploadResponse completedUploadResponse = s3Client.completeMultipartUpload( CompleteMultipartUploadRequest.builder().bucket(BUCKET).key(FILE_NAME).uploadId(createdUpload.uploadId()).multipartUpload(completedMultipartUpload).build()); Tag checksumTag = Tag.builder().key(CHECKSUM_TAG_KEYNAME).value(fullObjectChecksum).build(); //Optionally, if you need the full object checksum stored with the file; you could add it as a tag after completion. s3Client.putObjectTagging(PutObjectTaggingRequest.builder().bucket(BUCKET).key(FILE_NAME).tagging(Tagging.builder().tagSet(checksumTag).build()).build()); } catch (IOException | NoSuchAlgorithmException e) { e.printStackTrace(); } GetObjectAttributesResponse objectAttributes = s3Client.getObjectAttributes(GetObjectAttributesRequest.builder().bucket(BUCKET).key(FILE_NAME) .objectAttributes(ObjectAttributes.OBJECT_PARTS, ObjectAttributes.CHECKSUM).build()); System.out.println(objectAttributes.objectParts().parts()); System.out.println(objectAttributes.checksum().checksumSHA256()); } public static void downloadLargeFileBracketedByChecksum(S3Client s3Client) { System.out.println("Starting downloading file validation"); File file = new File("DOWNLOADED_" + FILE_NAME); try (OutputStream out = new FileOutputStream(file)) { GetObjectAttributesResponse objectAttributes = s3Client.getObjectAttributes(GetObjectAttributesRequest.builder().bucket(BUCKET).key(FILE_NAME) .objectAttributes(ObjectAttributes.OBJECT_PARTS, ObjectAttributes.CHECKSUM).build()); //Optionally if you need the full object checksum, you can grab a tag you added on the upload List objectTags = s3Client.getObjectTagging(GetObjectTaggingRequest.builder().bucket(BUCKET).key(FILE_NAME).build()).tagSet(); String fullObjectChecksum = null; for (Tag objectTag : objectTags) { if (objectTag.key().equals(CHECKSUM_TAG_KEYNAME)) { fullObjectChecksum = objectTag.value(); break; } } MessageDigest sha256FullObject = MessageDigest.getInstance("SHA-256"); MessageDigest sha256ChecksumOfChecksums = MessageDigest.getInstance("SHA-256"); //If you retrieve the object in parts, and set the ChecksumMode to enabled, the SDK will automatically validate the part checksum for (int partNumber = 1; partNumber <= objectAttributes.objectParts().totalPartsCount(); partNumber++) { MessageDigest sha256Part = MessageDigest.getInstance("SHA-256"); ResponseInputStream response = s3Client.getObject(GetObjectRequest.builder().bucket(BUCKET).key(FILE_NAME).partNumber(partNumber).checksumMode(ChecksumMode.ENABLED).build()); GetObjectResponse getObjectResponse = response.response(); byte[] buffer = new byte[CHUNK_SIZE]; int read = response.read(buffer); while (read != -1) { out.write(buffer, 0, read); sha256FullObject.update(buffer, 0, read); sha256Part.update(buffer, 0, read); read = response.read(buffer); } byte[] sha256PartBytes = sha256Part.digest(); sha256ChecksumOfChecksums.update(sha256PartBytes); //Optionally, you can do an additional manual validation again the part checksum if needed in addition to the SDK check String base64PartChecksum = Base64.getEncoder().encodeToString(sha256PartBytes); String base64PartChecksumFromObjectAttributes = objectAttributes.objectParts().parts().get(partNumber - 1).checksumSHA256(); if (!base64PartChecksum.equals(getObjectResponse.checksumSHA256()) || !base64PartChecksum.equals(base64PartChecksumFromObjectAttributes)) { throw new IOException("Part checksum didn't match for the part"); } System.out.println(partNumber + " " + base64PartChecksum); } //Before finalizing, do the final checksum validation. String base64FullObject = Base64.getEncoder().encodeToString(sha256FullObject.digest()); String base64ChecksumOfChecksums = Base64.getEncoder().encodeToString(sha256ChecksumOfChecksums.digest()); if (fullObjectChecksum != null && !fullObjectChecksum.equals(base64FullObject)) { throw new IOException("Failed checksum validation for full object"); } System.out.println(fullObjectChecksum); String base64ChecksumOfChecksumFromAttributes = objectAttributes.checksum().checksumSHA256(); if (base64ChecksumOfChecksumFromAttributes != null && !base64ChecksumOfChecksums.equals(base64ChecksumOfChecksumFromAttributes)) { throw new IOException("Failed checksum validation for full object checksum of checksums"); } System.out.println(base64ChecksumOfChecksumFromAttributes); out.flush(); } catch (IOException | NoSuchAlgorithmException e) { //Cleanup bad file file.delete(); e.printStackTrace(); } } public static void validateExistingFileAgainstS3Checksum(S3Client s3Client) { System.out.println("Starting existing file validation"); File file = new File("DOWNLOADED_" + FILE_NAME); GetObjectAttributesResponse objectAttributes = s3Client.getObjectAttributes(GetObjectAttributesRequest.builder().bucket(BUCKET).key(FILE_NAME) .objectAttributes(ObjectAttributes.OBJECT_PARTS, ObjectAttributes.CHECKSUM).build()); try (InputStream in = new FileInputStream(file)) { MessageDigest sha256ChecksumOfChecksums = MessageDigest.getInstance("SHA-256"); MessageDigest sha256Part = MessageDigest.getInstance("SHA-256"); byte[] buffer = new byte[CHUNK_SIZE]; int currentPart = 0; int partBreak = objectAttributes.objectParts().parts().get(currentPart).size(); int totalRead = 0; int read = in.read(buffer); while (read != -1) { totalRead += read; if (totalRead >= partBreak) { int difference = totalRead - partBreak; byte[] partChecksum; if (totalRead != partBreak) { sha256Part.update(buffer, 0, read - difference); partChecksum = sha256Part.digest(); sha256ChecksumOfChecksums.update(partChecksum); sha256Part.reset(); sha256Part.update(buffer, read - difference, difference); } else { sha256Part.update(buffer, 0, read); partChecksum = sha256Part.digest(); sha256ChecksumOfChecksums.update(partChecksum); sha256Part.reset(); } String base64PartChecksum = Base64.getEncoder().encodeToString(partChecksum); if (!base64PartChecksum.equals(objectAttributes.objectParts().parts().get(currentPart).checksumSHA256())) { throw new IOException("Part checksum didn't match S3"); } currentPart++; System.out.println(currentPart + " " + base64PartChecksum); if (currentPart < objectAttributes.objectParts().totalPartsCount()) { partBreak += objectAttributes.objectParts().parts().get(currentPart - 1).size(); } } else { sha256Part.update(buffer, 0, read); } read = in.read(buffer); } if (currentPart != objectAttributes.objectParts().totalPartsCount()) { currentPart++; byte[] partChecksum = sha256Part.digest(); sha256ChecksumOfChecksums.update(partChecksum); String base64PartChecksum = Base64.getEncoder().encodeToString(partChecksum); System.out.println(currentPart + " " + base64PartChecksum); } String base64CalculatedChecksumOfChecksums = Base64.getEncoder().encodeToString(sha256ChecksumOfChecksums.digest()); System.out.println(base64CalculatedChecksumOfChecksums); System.out.println(objectAttributes.checksum().checksumSHA256()); if (!base64CalculatedChecksumOfChecksums.equals(objectAttributes.checksum().checksumSHA256())) { throw new IOException("Full object checksum of checksums don't match S3"); } } catch (IOException | NoSuchAlgorithmException e) { e.printStackTrace(); } } } ``` ------ ### Using the REST API You can send REST requests to upload an object with a checksum value to verify the integrity of the data with [PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html). You can also retrieve the checksum value for objects using [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) or [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). ### Using the AWS CLI You can send a `PUT` request to upload an object of up to 5 GB in a single operation. For more information, see the [https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html#examples](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html#examples) in the *AWS CLI Command Reference*. You can also use [https://docs.aws.amazon.com/cli/latest/reference/s3api/get-object.html](https://docs.aws.amazon.com/cli/latest/reference/s3api/get-object.html) and [https://docs.aws.amazon.com/cli/latest/reference/s3api/head-object.html](https://docs.aws.amazon.com/cli/latest/reference/s3api/head-object.html) to retrieve the checksum of an already-uploaded object to verify the integrity of the data. For information, see [Amazon S3 CLI FAQ](https://docs.aws.amazon.com/cli/latest/topic/s3-faq.html) in the *AWS Command Line Interface User Guide*. ## Using Content-MD5 when uploading objects Another way to verify the integrity of your object after uploading is to provide an MD5 digest of the object when you upload it. If you calculate the MD5 digest for your object, you can provide the digest with the `PUT` command by using the `Content-MD5` header. After uploading the object, Amazon S3 calculates the MD5 digest of the object and compares it to the value that you provided. The request succeeds only if the two digests match. Supplying an MD5 digest isn't required, but you can use it to verify the integrity of the object as part of the upload process. ## Using Content-MD5 and the ETag to verify uploaded objects The entity tag (ETag) for an object represents a specific version of that object. Keep in mind that the ETag only reflects changes to the content of an object, not changes to its metadata. If only the metadata of an object changes, the ETag remains the same. Depending on the object, the ETag of the object might be an MD5 digest of the object data: + If an object is created by the `PutObject`, `PostObject`, or `CopyObject` operation, or through the AWS Management Console, and that object is also plaintext or encrypted by server-side encryption with Amazon S3 managed keys (SSE-S3), that object has an ETag that is an MD5 digest of its object data. + If an object is created by the `PutObject`, `PostObject`, or `CopyObject` operation, or through the AWS Management Console, and that object is encrypted by server-side encryption with customer-provided keys (SSE-C) or server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS), that object has an ETag that is not an MD5 digest of its object data. + If an object is created by either the multipart upload process or the `UploadPartCopy` operation, the object's ETag is not an MD5 digest, regardless of the method of encryption. If an object is larger than 16 MB, the AWS Management Console uploads or copies that object as a multipart upload, and therefore the ETag isn't an MD5 digest. For objects where the ETag is the `Content-MD5` digest of the object, you can compare the ETag value of the object with a calculated or previously stored `Content-MD5` digest. ## Using trailing checksums When uploading large objects to Amazon S3, you can either provide a precalculated checksum for the object or use an AWS SDK to automatically create trailing checksums for chunked uploads, on your behalf. If you use a trailing checksum, Amazon S3 automatically generates the checksum value by using your specified algorithm to validate the integrity of the object in chunked uploads, when you upload an object. To create a trailing checksum when using an AWS SDK, populate the `ChecksumAlgorithm` parameter with your preferred algorithm. The SDK uses that algorithm to calculate the checksum value for your object (or object parts) and automatically appends it to the end of your chunked upload request. This behavior saves you time because Amazon S3 performs both the verification and upload of your data in a single pass. **Important** If you're using S3 Object Lambda, all requests to S3 Object Lambda are signed using `s3-object-lambda` instead of `s3`. This behavior affects the signature of trailing checksum values. For more information about S3 Object Lambda, see [Transforming objects with S3 Object Lambda](transforming-objects.md). ### Trailing checksum headers To make a chunked content encoding request, Amazon S3 requires client servers to include several headers to correctly parse the request. Client servers must include the following headers: + **`x-amz-decoded-content-length`:**This header indicates the plaintext size of the actual data that is being uploaded to Amazon S3 with the request. + **`x-amz-content-sha256`:** This header indicates the type of chunked upload that is included in the request. For chunked uploads with trailing checksums, the header value is `STREAMING-UNSIGNED-PAYLOAD-TRAILER` for requests that don’t use payload signing and `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER` for requests that use SigV4 payload signing. (For more information about implementing signed payloads, see [Signature calculations for the authorization header: Transferring a payload in multiple chunks](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html).) + **`x-amz-trailer`:** This header indicates the name of the trailing header in the request. If trailing checksums exist (where AWS SDKs append checksums to the encoded request bodies), the `x-amz-trailer` header value includes the `x-amz-checksum-` prefix and ends with the algorithm name. The following `x-amz-trailer` values are currently supported: + `x-amz-checksum-crc32` + `x-amz-checksum-crc32c` + `x-amz-checksum-crc64nvme` + `x-amz-checksum-sha1` + `x-amz-checksum-sha256` **Note** You can also include the `Content-Encoding` header, with the chunked value, in your request. While this header isn’t required, including this header can minimize HTTP proxy issues when transmitting encoded data. If another `Content-Encoding` header (such as gzip) exists in the request, the `Content-Encoding` header includes the chunked value in a comma-separated list of encodings. For example, `Content-Encoding: aws-chunked, gzip`. ### Chunked parts When you upload an object to Amazon S3 using chunked encoding, the upload request includes the following types of chunks (formatted in the listed order): + **Object body chunks:** There can be one, multiple, or zero body chunks associated with a chunked upload request. + **Completion chunks: ** There can be one, multiple, or zero body chunks associated with a chunked upload request. + **Trailing chunks:** The trailing checksum is listed after the completion chunk. Only one trailing chunk is allowed. **Note** Every chunked upload must end with a final CRLF (such as `\r\n`) to indicate the end of the request. For examples of chunked formatting, see [Examples: Chunked uploads with trailing checksums](#example-chunked-uploads-trailing). #### Object body chunks Object body chunks are the chunks that contain the actual object data that is being uploaded to S3. These chunks have consistent size and format constraints. ##### Object body chunk size These chunks must contain at least 8,192 bytes (or 8 KiB) of object data, except for the final body chunk, which can be smaller. There is no explicit maximum chunk size but you can expect all chunks to be smaller than the 5 GB maximum upload size. Chunk sizes can vary from one chunk to the next based on your client server implementation. ##### Object body chunk format Object body chunks begin with the hexadecimal encoding of the number of bytes in the object body chunk, followed by a CRLF (Carriage Return Line Feed), the object bytes for that chunk, and another CRLF. For example: ``` hex-encoding-of-object-bytes-in-chunk\r\n chunk-object-bytes\r\n ``` However, when the chunk is signed, the object body chunk follows a different format, where the signature is appended to the chunk size with a semicolon delimiter. For example: ``` hex-encoding-of-object-bytes-in-chunk;chunk-signature\r\n chunk-object-bytes\r\n ``` For more information about chunk signing, see [Signature calculations for the Authorization Header: Transferring a payload in multiple chunks (AWS Signature Version 4)](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). For more information about chunk formatting, see [Chunked transfer encoding](https://www.rfc-editor.org/rfc/rfc9112#name-chunked-transfer-coding) on the *RFC Editor* website. #### Completion chunks Completion chunks are required to be the final object body chunk of every chunked upload. A completion chunk's format is similar to a body chunk, but always contains zero bytes of object data. (The zero bytes of object data indicate that all of the data has been uploaded.) Chunked uploads must include a completion chunk as its final object body chunk, following a format like this: ``` 0\r\n ``` However, if the content encoding request uses payload signing, it follows this format instead: ``` 0;chunk-signature\r\n ``` #### Trailer chunks Trailer chunks hold the calculated checksum for all S3 upload requests. Trailer chunks include two fields: one header name field and one header value field. The header name field for an upload request must match the value passed into the `x-amz-trailer` request header. For example, if a request contains `x-amz-trailer: x-amz-checksum-crc32` and the trailer chunk has the header name `x-amz-checksum-sha1`, the request fails. The value field in the trailer chunk includes a base64 encoding of the big-endian checksum value for that object. (The big-endian ordering stores the most significant byte of data at the lowest memory address, and the least significant byte at the largest memory address). The algorithm used to calculate this checksum is the same as the suffix for the header name (for example, `crc32`). ##### Trailer chunk format Trailer chunks use the following format for unsigned payload requests: ``` x-amz-checksum-lowercase-checksum-algorithm-name:base64-checksum-value\n\r\n\r\n ``` For requests with [SigV4 signed payloads](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html), the trailer chunk includes a trailer signature after the trailer chunk. ``` trailer-checksum\n\r\n trailer-signature\r\n ``` You can also add the CRLF directly to the end of the base64 checksum value. For example: ``` x-amz-checksum-lowercase-checksum-algorithm-name:base64-checksum-value\r\n\r\n ``` #### Examples: Chunked uploads with trailing checksums Amazon S3 supports chunked uploads that use `aws-chunked` content encoding for `PutObject` and `UploadPart` requests with trailing checksums. **Example 1 – Unsigned chunked `PutObject` request with a trailing CRC-32 checksum** The following is an example of a chunked `PutObject` request with a trailing CRC-32 checksum. In this example, the client uploads a 17 KB object in three unsigned chunks and appends a trailing CRC-32 checksum chunk by using the `x-amz-checksum-crc32` header. ``` PUT /Key+ HTTP/1.1 Host: amzn-s3-demo-bucket Content-Encoding: aws-chunked x-amz-decoded-content-length: 17408 x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER x-amz-trailer: x-amz-checksum-crc32 2000\r\n // Object body chunk 1 (8192 bytes) object-bytes\r\n 2000\r\n // Object body chunk 2 (8192 bytes) object-bytes\r\n 400\r\n // Object body chunk 3 (1024 bytes) object-bytes\r\n 0\r\n // Completion chunk x-amz-checksum-crc32:YABb/g==\n\r\n\r\n // Trailer chunk (note optional \n character) \r\n // CRLF ``` Here's an example response: ``` HTTP/1.1 200 ETag: ETag x-amz-checksum-crc32: YABb/g== ``` **Note** The usage of the linefeed `\n` at the end of the checksum value might vary across clients. **Example 2 – SigV4-signed chunked `PutObject` request with a trailing CRC-32 (`CRC32`) checksum** The following is an example of a chunked `PutObject` request with a trailing CRC-32 checksum. This request uses SigV4 payload signing. In this example, the client uploads a 17 KB object in three signed chunks. In addition to the `object body` chunks, the `completion chunk` and `trailer chunk` are also signed. ``` PUT /Key+ HTTP/1.1 Host: amzn-s3-demo-bucket.s3.amazonaws.com Content-Encoding: aws-chunked x-amz-decoded-content-length: 17408 x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER x-amz-trailer: x-amz-checksum-crc32 authorization-code // SigV4 headers authorization 2000;chunk-signature=signature-value...\r\n // Object body chunk 1 (8192 bytes) object-bytes\r\n 2000;chunk-signature\r\n // Object body chunk 2 (8192 bytes) object-bytes\r\n 400;chunk-signature\r\n // Object body chunk 3 (1024 bytes) object-bytes\r\n 0;chunk-signature\r\n // Completion chunk x-amz-checksum-crc32:YABb/g==\n\r\n // Trailer chunk (note optional \n character) trailer-signature\r\n \r\n // CRLF ``` Here's an example response: ``` HTTP/1.1 200 ETag: ETag x-amz-checksum-crc32: YABb/g== ``` # Checking object integrity for data at rest in Amazon S3 If you need to verify the content of datasets stored in Amazon S3, the S3 Batch Operations [Compute checksum](https://docs.aws.amazon.com//AmazonS3/latest/userguide/batch-ops-compute-checksums.html) operation calculates both full object or composite checksums for objects at rest. The **Compute checksum** operation uses Batch Operations to asynchronously calculate the checksum values for a group of objects and automatically generates a consolidated integrity report, without creating new copies of your data, or restoring or downloading any data. With the **Compute checksum** operation, you can efficiently verify billions of objects with a single job request. For each **Compute checksum** job request, S3 calculates the checksum values, and includes it in an automatically generated integrity report (also known as a completion report). You can then use the completion report to validate the integrity of your dataset. The **Compute checksum** operation works with any object stored in S3, regardless of storage class or object size. Whether you need to verify your objects as a data preservation best practice, or meet compliance requirements, the **Compute checksum** operation reduces the cost, time, and effort required for data validation by performing checksum calculations at rest. For information about **Compute checksum** pricing, see the **Management and insights** tab on [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/). Then, you can use the output of the generated completion report to compare against the checksum values that you’ve stored in your databases to verify that your datasets remain intact over time. This approach helps you maintain end-to-end data integrity for business and compliance needs. For example, you can use the **Compute checksum** operation to submit a list of stored objects in S3 Glacier storage classes for annual security audits. Additionally, the range of supported checksum algorithms allow you to maintain continuity with the algorithms that are used in your applications. ## Using supported checksum algorithms For data at rest, you can calculate checksums, using any of the supported checksum algorithms: + CRC-64/NVME (`CRC64NVME`): Supports the full object checksum type only. + CRC-32 (`CRC32`): Supports both full object and composite checksum types. + CRC-32C (`CRC32C`): Supports both full object and composite checksum types. + SHA-1 (`SHA1`): Supports both full object and composite checksum types. + SHA-256 (`SHA256`): Supports both full object and composite checksum types. + MD5 (`MD5`): Supports both full object and composite checksum types. ## Using **Compute checksum** For objects stored in Amazon S3, you can use the **Compute checksum** operation with S3 Batch Operations to check the content of stored data at rest. You can [create a Compute checksum Batch Operations job](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html) by using the Amazon S3 console, AWS Command Line Interface (AWS CLI), REST API, or AWS SDK. When the **Compute checksum** job finishes, you receive a completion report. For more information about how to use the completion report, see [Tracking job status and completion reports](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-job-status.html). Before creating your **Compute checksum** job, you must create an S3 Batch Operations AWS Identity and Access Management (IAM) role to grant Amazon S3 permissions to perform actions on your behalf. You’ll need to grant permissions to read the manifest file and to write a completion report to the S3 bucket. For more information, see [Compute checksums](batch-ops-compute-checksums.md). ### Using the S3 console **To use the **Compute checksum** operation** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the navigation bar on the top of the page, choose the name of the currently displayed AWS Region. Next, choose the Region in which you want to create your job. **Note** For copy operations, you must create the job in the same Region as the destination bucket. For all other operations, you must create the job in the same Region as the objects in the manifest. 1. Choose **Batch Operations** on the left navigation pane of the Amazon S3 console. 1. Choose **Create job**. 1. View the AWS Region where you want to create your job. **Note** For copy operations, you must create the job in the same Region as the destination bucket. For all other operations, you must create the job in the same Region as the objects in the manifest. 1. Under **Manifest format**, choose the type of manifest object to use. + If you choose **S3 inventory report (manifest.json)**, enter the path to the `manifest.json` object, and (optionally) the **Manifest object version ID** if you want to use a specific object version. Alternatively, you can choose **Browse S3** and choose the manifest JSON file, which auto populates all the manifest object field entries. + If you choose **CSV**, choose the Manifest location type, Then, either enter the path to a CSV-formatted manifest object or choose **Browse S3** to select a manifest object. The manifest object must follow the format described in the console. If you want to use a specific version of the manifest object, then you can also specify the object version ID. + If you choose **Create manifest using S3 Replication configuration**, a list of objects will be generated using the replication configuration and optionally saved to the destination you choose. When using a replication configuration to generate the manifest, the only operation that will be available is **Replicate**. 1. Choose **Next**. 1. Under **Operation**, choose the **Compute checksum** operation to calculate the checksums on all objects listed in the manifest. Choose the **Checksum type** and **Checksum function** for your job. Then, choose **Next**. 1. Fill out the information for **Configure additional options**, and then choose **Next**. 1. On the **Configure additional options** page, fill out the information for your **Compute checksum** job. **Note** Under **Completion report**, make sure to confirm the acknowledgement statement. This acknowledgement statement confirms that you understand that the completion report contains checksum values, which are used to verify the integrity of data stored in Amazon S3. Therefore, the completion report should be shared with caution. Also, be aware that if you're creating a Compute checksum request and you specify an external account owner's bucket location to store your completion report, make sure to specify the AWS account ID of the external bucket owner. 1. Choose **Next**. 1. On the **Review** page, review and confirm your settings. 1. (Optional) If you need to make changes, choose **Previous** to go back to the previous page, or choose **Edit** to update a specific step. 1. After you've confirmed your changes, choose **Create job**. **To list and monitor the progress of all **Compute checksum** requests** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Batch Operations**. 1. On the **Batch Operations** page, you can review job details such as the job priority, job completion rate, and total objects. 1. If you want to manage or clone a specific **Compute checksum** job, click on the **Job ID** to review additional job information. 1. On the specific **Compute checksum** job page, review the job details. Each batch operations job progresses through different [job statuses](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-job-status.html#batch-ops-job-status-table). You can also [enable AWS CloudTrail events](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-cloudtrail-logging-for-s3.html) in the S3 console to receive alerts on any job state changes. For active jobs, you can review the running job and completion rate on the **Job details** page. ### Using the AWS SDKs ------ #### [ Java ] **Example: Creating a **Compute checksum** job** The following example shows you how to create a **Compute checksum** job (as part of a **Create job** request), and how to specify a manifest: ``` // Required parameters String accountId = "111122223333"; String roleArn = "arn:aws:iam::111122223333:role/BatchOperations"; String manifestArn = "arn:aws:s3:::my_manifests/manifest.csv"; String manifestEtag = "60e460c9d1046e73f7dde5043ac3ae85"; String reportBucketArn = "arn:aws:s3:::amzn-s3-demo-completion-report-bucket"; String reportExpectedBucketOwner = "111122223333"; String reportPrefix = "demo-report"; // Job Operation S3ComputeObjectChecksumOperation s3ComputeObjectChecksum = S3ComputeObjectChecksumOperation.builder() .checksumAlgorithm(ComputeObjectChecksumAlgorithm.CRC64) .checksumType(ComputeObjectChecksumType.COMPOSITE) .build(); JobOperation operation = JobOperation.builder() .s3ComputeObjectChecksum(s3ComputeObjectChecksum) .build(); // Job Manifest JobManifestLocation location = JobManifestLocation.builder() .eTag(manifestEtag) .objectArn(manifestArn) .build(); JobManifestSpec spec = JobManifestSpec.builder() .format(JobManifestFormat.S3_BATCH_OPERATIONS_CSV_20180820) .fields(Arrays.asList(JobManifestFieldName.BUCKET, JobManifestFieldName.KEY)) .build(); JobManifest manifest = JobManifest.builder() .location(location) .spec(spec) .build(); // Completion Report JobReport report = JobReport.builder() .bucket(reportBucketArn) .enabled(true) // Must be true .expectedBucketOwner(reportExpectedBucketOwner) .format(JobReportFormat.REPORT_CSV_20180820) .prefix(reportPrefix) .reportScope(JobReportScope.ALL_TASKS) .build(); // Create Job Request CreateJobRequest request = CreateJobRequest.builder() .accountId(accountId) .confirmationRequired(false) .manifest(manifest) .operation(operation) .priority(10) .report(report) .roleArn(roleArn); // Create the client S3ControlClient client = S3ControlClient.builder() .credentialsProvider(new ProfileCredentialsProvider()) .region(Region.US_EAST_1) .build(); // Send the request try { CreateJobResponse response = client.createJob(request); System.out.println(response); } catch (AwsServiceException e) { System.out.println("AwsServiceException: " + e.getMessage()); throw new RuntimeException(e); } catch (SdkClientException e) { System.out.println("SdkClientException: " + e.getMessage()); throw new RuntimeException(e); } ``` **Example: Viewing **Compute checksum** job details** The following example shows how you can specify a job ID to view the job details (such as the job completion rate) for a **Compute checksum** job request: ``` DescribeJobRequest request = DescribeJobRequest.builder() .accountId("1234567890") .jobId("a16217a1-e082-48e5-b04f-31fac3a66b13") .build(); ``` ------ ### Using the AWS CLI You can use the [https://docs.aws.amazon.com/cli/latest/reference/s3control/create-job.html](https://docs.aws.amazon.com/cli/latest/reference/s3control/create-job.html) command to create a new batch operations job, and to provide the list of objects. Then, specify the checksum algorithm and checksum type, and the destination bucket where you want to save the **Compute checksum** report. The following example creates an S3 Batch Operations **Compute checksum** job by using an S3 generated manifest for the AWS account *111122223333*. To use this command, replace the *user input placeholders* with your own information: ``` aws s3control create-job \ --account-id 111122223333 \ --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::my-manifest-bucket/manifest.csv","ETag":"e0e8bfc50e0f0c5d5a1a5f0e0e8bfc50"}}' \ --manifest-generator '{ "S3JobManifestGenerator": { "ExpectedBucketOwner": "111122223333", "SourceBucket": "arn:aws:s3:::amzn-s3-demo-source-bucket", "EnableManifestOutput": true, "ManifestOutputLocation": { "ExpectedManifestBucketOwner": "111122223333", "Bucket": "arn:aws:s3:::amzn-s3-demo-manifest-bucket", "ManifestPrefix": "prefix", "ManifestFormat": "S3InventoryReport_CSV_20211130" }, "Filter": { "CreatedAfter": "2023-09-01", "CreatedBefore": "2023-10-01", "KeyNameConstraint": { "MatchAnyPrefix": [ "prefix" ], "MatchAnySuffix": [ "suffix" ] }, "ObjectSizeGreaterThanBytes": 100, "ObjectSizeLessThanBytes": 200, "MatchAnyStorageClass": [ "STANDARD", "STANDARD_IA" ] } } }' \ --operation '{"S3ComputeObjectChecksum":{"ChecksumAlgorithm":"CRC64NVME","ChecksumType":"FULL_OBJECT"}}' \ --report '{"Bucket":"arn:aws:s3:::my-report-bucket","Format":"Report_CSV_20180820","Enabled":true,"Prefix":"batch-op-reports/","ReportScope":"AllTasks","ExpectedBucketOwner":"111122223333"}' \ --priority 10 \ --role-arn arn:aws:iam::123456789012:role/S3BatchJobRole \ --client-request-token 6e023a7e-4820-4654-8c81-7247361aeb73 \ --description "Compute object checksums" \ --region us-west-2 ``` After you submit the **Compute checksum** job, you receive the job ID as a response and it appears on the S3 Batch Operations list page. Amazon S3 processes the list of objects and calculates checksums for each object. After the job finishes, S3 provides a consolidated **Compute checksum** report at the specified destination. To monitor the progress of your **Compute checksum** job, use the [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3control/describe-job.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3control/describe-job.html) command. This command checks the status of the specified batch operations job. To use this command, replace the *user input placeholders* with your own information. For example: ``` aws s3control describe-job --account-id 111122223333 --job-id 1234567890abcdef0 ``` To [obtain a list](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-list-jobs.html) of all **Active** and **Complete** batch operations jobs, see [Listing jobs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-list-jobs.html) or [list-jobs](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3control/list-jobs.html) in the *AWS CLI Command Reference*. ### Using the REST API You can send REST requests to verify object integrity with **Compute checksum** using [CreateJob](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html). You can monitor the progress of **Compute checksum** requests by sending REST requests to the [DescribeJob](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_DescribeJob.html) API operation. Each batch operations job progresses through the following statuses: + **NEW** + **PREPARING** + **READY** + **ACTIVE** + **PAUSING** + **PAUSED** + **COMPLETE** + **CANCELLING** + **FAILED** The API response notifies you of the current job state. # Deleting Amazon S3 objects You can delete one or more objects directly from Amazon S3 using the Amazon S3 console, AWS SDKs, AWS Command Line Interface (AWS CLI), or REST API. For example, if you're collecting log files, it's a good idea to delete them when they're no longer needed. You can [set up an S3 Lifecycle rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html) to automatically delete objects such as log files. To delete an object, you can use one of the following API operations: + **Delete a single object** – Amazon S3 provides the `DELETE` (`DeleteObject`) API operation that you can use to delete one object in a single HTTP request. + **Delete multiple objects** – Amazon S3 provides the Multi-Object Delete (`DeleteObjects`) API operation that you can use to delete up to 1,000 objects in a single HTTP request. When deleting objects from a bucket that is not versioning-enabled, you provide only the object key name. However, when deleting objects from a versioning-enabled bucket, you can provide the version ID of the object to delete a specific version of the object. ## Best practices to consider before deleting an object Before you delete an object, consider the following best practices: + Enable [bucket versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html). [S3 Versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html) adds protection against simple `DeleteObject` requests to prevent accidental deletions. For versioned buckets, if you delete the current version of an object or when a delete request doesn’t specify a specific version Id, Amazon S3 doesn’t permanently delete the object. Instead, S3 adds a delete marker, issuing a soft delete of the object. The delete marker then becomes the current (or newest) version of the object with a new version ID. For more information, see [Deleting object versions from a versioning-enabled bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html). + If you want to delete a large number of objects, or for programmatically deleting objects based on object creation date, [set a S3 Lifecycle configuration on your bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html). To monitor these deletions, we recommend that you [use an S3 Lifecycle event notification](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configure-notification.html). When you configure S3 Lifecycle notifications, the `s3:LifecycleExpiration:Delete` event type notifies you when an object in a bucket is deleted. It also notifies you when an object version is permanently deleted by an S3 Lifecycle configuration. The `s3:LifecycleExpiration:DeleteMarkerCreated` event type notifies you when S3 Lifecycle creates a delete marker. A delete marker is created when a current version of an object in a versioned bucket is deleted. + Before making any updates to your S3 Lifecycle configuration, confirm that Lifecycle has completed the actions on all intended objects. For more information, see the **Updating, disabling, or deleting Lifecycle rules** section in [Setting an S3 Lifecycle configuration on a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html). **Note** The S3 Lifecycle rules must apply to the right subset of objects to prevent unintended deletions. You can filter objects by prefixes, object tags, or object sizes when creating the Lifecycle rules. + Consider restricting users from removing or deleting objects from your bucket. To restrict users, you’ll need to explicitly deny users the permissions for the following actions in your [Amazon S3 bucket policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html): + `s3:DeleteObject`, `s3:DeleteObjectVersion` (to control who can delete objects using API requests) + `s3:PutLifecycleConfiguration` (to control who can add S3 Lifecycle expiration rules) + Consider using [S3 Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) to create multiple copies of your data and to replicate them to multiple locations at once. You can choose as many destination buckets as needed. Additionally, if an object is unintentionally deleted, you'll still have a copy of the data. + Avoid sending object version IDs to unversioned buckets. Also, make sure to properly set both the `s3:DeleteObject` and `s3:DeleteObjectVersion` permissions on all buckets (including unversioned buckets). ## Deleting objects from a versioning-enabled bucket If your bucket is versioning-enabled, multiple versions of the same object can exist in the bucket. When working with versioning-enabled buckets, the `Delete` API operations enable the following options: + **Specify a non-versioned delete request** – Specify only the object's key, and not the version ID. In this case, Amazon S3 creates a delete marker over the current version of the object and returns its version ID in the response. This makes your object disappear from the bucket. For information about object versioning and the delete marker concept, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). + **Specify a versioned delete request** – Specify both the key and version ID. In this case, the following outcomes are possible: + If the version ID maps to a specific object version, Amazon S3 deletes the specific version of the object. + If the version ID maps to the delete marker of an object, Amazon S3 deletes the delete marker. When the delete marker gets deleted, the object then reappears in your bucket. ## Deleting objects from a versioning-suspended bucket If your bucket is versioning-suspended, the `Delete` API operations behave the same way for versioning enabled buckets (except for when the current version has a null version ID). For more information, see [Deleting objects from versioning-suspended buckets](DeletingObjectsfromVersioningSuspendedBuckets.md). ## Deleting objects from an unversioned bucket If your bucket is unversioned, you can specify the object's key in the `Delete` API operations and Amazon S3 will permanently delete the object. To prevent permanent deletion of an object, [enable bucket versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html). For unversioned buckets, if the `s3:DeleteObject` or `s3:DeleteObjectVersion` permissions are explicitly denied in your bucket policy, then any `DeleteObject`, `DeleteObjects`, or `DeleteObjectVersion` requests result in a `403 Access Denied` error. ## Deleting objects from an MFA-enabled bucket When deleting objects from a multi-factor authentication (MFA)-enabled bucket, note the following: + If you provide an MFA token that isn't valid, the request always fails. + If you have an MFA-enabled bucket and you make a versioned delete request (you provide an object key and version ID), the request fails if you don't provide a valid MFA token. In addition, when using the multi-object `Delete` API operation on an MFA-enabled bucket, if any of the deletes are a versioned delete request (that is, you specify an object key and version ID), the entire request fails if you don't provide an MFA token. However, in the following cases, the request succeeds: + If you have an MFA-enabled bucket and you make a non-versioned delete request (you are not deleting a versioned object), and you don't provide an MFA token, the delete succeeds. + If you have a Multi-Object Delete request that specifies only non-versioned objects to delete from an MFA-enabled bucket and you don't provide an MFA token, the deletions succeed. For information about MFA delete, see [Configuring MFA delete](MultiFactorAuthenticationDelete.md). **Topics** + [ ## Best practices to consider before deleting an object ](#DeletingObjects-best-practices) + [ ## Deleting objects from a versioning-enabled bucket ](#DeletingObjectsfromaVersion-EnabledBucket) + [ ## Deleting objects from a versioning-suspended bucket ](#Deleting-objects-versioning-suspended-bucket) + [ ## Deleting objects from an unversioned bucket ](#Deleting-objects-unversioned-bucket) + [ ## Deleting objects from an MFA-enabled bucket ](#DeletingObjectsfromanMFA-EnabledBucket) + [ # Deleting a single object ](delete-objects.md) + [ # Deleting multiple objects ](delete-multiple-objects.md) # Deleting a single object You can use the Amazon S3 console or the DELETE API to delete a single existing object from an S3 bucket. For more information about deleting objects in Amazon S3, see [Deleting Amazon S3 objects](DeletingObjects.md). Because all objects in your S3 bucket incur storage costs, you should delete objects that you no longer need. For example, if you are collecting log files, it's a good idea to delete them when they're no longer needed. You can set up a lifecycle rule to automatically delete objects such as log files. For more information, see [Setting an S3 Lifecycle configuration on a bucket](how-to-set-lifecycle-configuration-intro.md). For information about Amazon S3 features and pricing, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing). ## Using the S3 console Follow these steps to use the Amazon S3 console to delete a single object from a bucket. **Warning** When you permanently delete an object or specified object version in the Amazon S3 console, the deletion can't be undone. **To delete an object that has versioning enabled or suspended** **Note** If the version ID for an object in a versioning-suspended bucket is marked as `NULL`, S3 permanently deletes the object since no previous versions exist. However, if a valid version ID is listed for the object in a versioning-suspended bucket, then S3 creates a delete marker for the deleted object, while retaining the previous versions of the object. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets** or **Directory buckets**. 1. In the bucket list, choose the name of the bucket that you want to delete an object from. 1. Select the object and then choose **Delete**. 1. To confirm deletion of the objects list under **Specified objects** in the **Delete objects?** text box, enter **delete**. **To permanently delete a specific object version in a versioning-enabled bucket** **Warning** When you permanently delete a specific object version in Amazon S3, the deletion can't be undone. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the **Bucket name** list, choose the name of the bucket that you want to delete an object from. 1. Select the object that you want to delete. 1. Choose the **Show versions** toggle. 1. Select the object version and then choose **Delete**. 1. To confirm permanent deletion of the specific object versions listed under **Specified objects**, in the **Delete objects?** text box, enter **Permanently delete**. Amazon S3 permanently deletes the specific object version. **To permanently delete an object in an Amazon S3 bucket that *doesn't* have versioning enabled** **Warning** When you permanently delete an object in Amazon S3, the deletion can't be undone. Also, for any buckets without versioning enabled, deletions are permanent. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the bucket list, choose the name of the bucket that you want to delete an object from. 1. Select the object and then choose **Delete**. 1. To confirm permanent deletion of the object listed under **Specified objects**, in the **Delete objects?** text box, enter **permanently delete**. **Note** If you're experiencing any issues with deleting your object, see [I want to permanently delete versioned objects](troubleshooting-versioning.md#delete-objects-permanent). ## Using the AWS CLI To delete one object per request, use the `DELETE` API. For more information, see [DELETE Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETE.html). For more information about using the CLI to delete an object, see [delete-object](https://awscli.amazonaws.com/v2/documentation/api/2.0.34/reference/s3api/delete-object.html). # Using the REST API You can use the AWS SDKs to delete an object. However, if your application requires it, you can send REST requests directly. For more information, see [DELETE Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETE.html) in the *Amazon Simple Storage Service API Reference*. ## Using the AWS SDKs The following examples show how you can use the AWS SDKs to delete an object from a bucket. For more information, see [DELETE Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETE.html) in the *Amazon Simple Storage Service API Reference* If you have S3 Versioning enabled on the bucket, you have the following options: + Delete a specific object version by specifying a version ID. + Delete an object without specifying a version ID, in which case Amazon S3 adds a delete marker to the object. For more information about S3 Versioning, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). For more examples, and examples in other languages, see [Use `DeleteObject` with an AWS SDK or CLI ](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_DeleteObject_section.html) in the *Amazon S3 API reference*. ------ #### [ Java ] To delete objects using the AWS SDK for Java, you can delete objects from both versioned and non-versioned buckets: + *Non-versioned bucket:* In the delete request, you specify only the object key and not a version ID. + *Versioned bucket:* You can delete a specific object version by specifying both the object key name and version ID. If there are no other versions of that object, Amazon S3 deletes the object entirely. Otherwise, Amazon S3 only deletes the specified version. **Note** You can get the version IDs of an object by sending a `ListVersions` request. For examples of how to delete a single object with the AWS SDK for Java, see [Delete an object](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_DeleteObject_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] The following examples show how to delete an object from both versioned and non-versioned buckets. For more information about S3 Versioning, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). **Example Deleting an object from a non-versioned bucket** The following C\$1 example deletes an object from a non-versioned bucket. The example assumes that the objects don't have version IDs, so you don't specify version IDs. You specify only the object key. For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Model; using System; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class DeleteObjectNonVersionedBucketTest { private const string bucketName = "*** bucket name ***"; private const string keyName = "*** object key ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 client; public static void Main() { client = new AmazonS3Client(bucketRegion); DeleteObjectNonVersionedBucketAsync().Wait(); } private static async Task DeleteObjectNonVersionedBucketAsync() { try { var deleteObjectRequest = new DeleteObjectRequest { BucketName = bucketName, Key = keyName }; Console.WriteLine("Deleting an object"); await client.DeleteObjectAsync(deleteObjectRequest); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when deleting an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when deleting an object", e.Message); } } } } ``` **Example Deleting an object from a versioned bucket** The following C\$1 example deletes an object from a versioned bucket. It deletes a specific version of the object by specifying the object key name and version ID. The code performs the following tasks: 1. Enables S3 Versioning on a bucket that you specify (if S3 Versioning is already enabled, this has no effect). 1. Adds a sample object to the bucket. In response, Amazon S3 returns the version ID of the newly added object. The example uses this version ID in the delete request. 1. Deletes the sample object by specifying both the object key name and a version ID. **Note** You can also get the version ID of an object by sending a `ListVersions` request. ``` var listResponse = client.ListVersions(new ListVersionsRequest { BucketName = bucketName, Prefix = keyName }); ``` For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Model; using System; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { class DeleteObjectVersion { private const string bucketName = "*** versioning-enabled bucket name ***"; private const string keyName = "*** Object Key Name ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 client; public static void Main() { client = new AmazonS3Client(bucketRegion); CreateAndDeleteObjectVersionAsync().Wait(); } private static async Task CreateAndDeleteObjectVersionAsync() { try { // Add a sample object. string versionID = await PutAnObject(keyName); // Delete the object by specifying an object key and a version ID. DeleteObjectRequest request = new DeleteObjectRequest { BucketName = bucketName, Key = keyName, VersionId = versionID }; Console.WriteLine("Deleting an object"); await client.DeleteObjectAsync(request); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when deleting an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when deleting an object", e.Message); } } static async Task PutAnObject(string objectKey) { PutObjectRequest request = new PutObjectRequest { BucketName = bucketName, Key = objectKey, ContentBody = "This is the content body!" }; PutObjectResponse response = await client.PutObjectAsync(request); return response.VersionId; } } } ``` ------ #### [ PHP ] This example shows how to use classes from version 3 of the AWS SDK for PHP to delete an object from a non-versioned bucket. For information about deleting an object from a versioned bucket, see [Using the REST API](DeletingAnObjectsUsingREST.md). For more information about the AWS SDK for Ruby API, go to [AWS SDK for Ruby - Version 2](https://docs.aws.amazon.com/sdkforruby/api/index.html). The following PHP example deletes an object from a bucket. Because this example shows how to delete objects from non-versioned buckets, it provides only the bucket name and object key (not a version ID) in the delete request. ``` 'latest', 'region' => 'us-east-1' ]); // 1. Delete the object from the bucket. try { echo 'Attempting to delete ' . $keyname . '...' . PHP_EOL; $result = $s3->deleteObject([ 'Bucket' => $bucket, 'Key' => $keyname ]); if ($result['DeleteMarker']) { echo $keyname . ' was deleted or does not exist.' . PHP_EOL; } else { exit('Error: ' . $keyname . ' was not deleted.' . PHP_EOL); } } catch (S3Exception $e) { exit('Error: ' . $e->getAwsErrorMessage() . PHP_EOL); } // 2. Check to see if the object was deleted. try { echo 'Checking to see if ' . $keyname . ' still exists...' . PHP_EOL; $result = $s3->getObject([ 'Bucket' => $bucket, 'Key' => $keyname ]); echo 'Error: ' . $keyname . ' still exists.'; } catch (S3Exception $e) { exit($e->getAwsErrorMessage()); } ``` ------ #### [ Javascript ] ``` import { DeleteObjectCommand } from "@aws-sdk/client-s3"; import { s3Client } from "./libs/s3Client.js" // Helper function that creates Amazon S3 service client module. export const bucketParams = { Bucket: "BUCKET_NAME", Key: "KEY" }; export const run = async () => { try { const data = await s3Client.send(new DeleteObjectCommand(bucketParams)); console.log("Success. Object deleted.", data); return data; // For unit tests. } catch (err) { console.log("Error", err); } }; run(); ``` ------ # Deleting multiple objects Because all objects in your S3 bucket incur storage costs, you should delete objects that you no longer need. For example, if you are collecting log files, it's a good idea to delete them when they're no longer needed. You can set up a lifecycle rule to automatically delete objects such as log files. For more information, see [Setting an S3 Lifecycle configuration on a bucket](how-to-set-lifecycle-configuration-intro.md). For information about Amazon S3 features and pricing, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing). You can use the Amazon S3 console, AWS SDKs, or the REST API to delete multiple objects simultaneously from an S3 bucket. ## Using the S3 console Follow these steps to use the Amazon S3 console to delete multiple objects from a bucket. **Warning** Deleting a specified object cannot be undone. This action deletes all specified objects. When deleting folders, wait for the delete action to finish before adding new objects to the folder. Otherwise, new objects might be deleted as well. When deleting objects in a bucket without versioning enabled, including directory buckets, Amazon S3 will permanently delete the objects. When deleting objects in a bucket with bucket versioning **enabled** or **suspended**, Amazon S3 creates delete markers. For more information, see [Working with delete markers](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html). **To delete objects that have versioning enabled or suspended** **Note** If the version IDs for the object in a versioning-suspended bucket are marked as `NULL`, S3 permanently deletes the objects since no previous versions exist. However, if a valid version ID is listed for the objects in a versioning-suspended bucket, then S3 creates delete markers for the deleted objects, while retaining the previous versions of the objects. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the bucket list, choose the name of the bucket that you want to delete the objects from. 1. Select the objects and then choose **Delete**. 1. To confirm deletion of the objects list under **Specified objects** in the **Delete objects?** text box, enter **delete**. **To permanently delete specific object versions in a versioning-enabled bucket** **Warning** When you permanently delete specific object versions in Amazon S3, the deletion can't be undone. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the bucket list, choose the name of the bucket that you want to delete the objects from. 1. Select the objects that you want to delete. 1. Choose the **Show versions** toggle. 1. Select the object versions and then choose **Delete**. 1. To confirm permanent deletion of the specific object versions listed under **Specified objects**, in the **Delete objects?** text box, enter **Permanently delete**. Amazon S3 permanently deletes the specific object versions. **To permanently delete the objects in an Amazon S3 bucket that *don't* have versioning enabled** **Warning** When you permanently delete an object in Amazon S3, the deletion can't be undone. Also, for any buckets without versioning enabled, including directory buckets, deletions are permanent. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets** or **Directory buckets**. 1. In the bucket list, choose the name of the bucket that you want to delete the objects from. 1. Select the objects and then choose **Delete**. 1. To confirm permanent deletion of the objects listed under **Specified objects**, in the **Delete objects?** text box, enter **permanently delete**. **Note** If you're experiencing any issues with deleting your objects, see [I want to permanently delete versioned objects](troubleshooting-versioning.md#delete-objects-permanent). ## Using the AWS SDKs For examples of how to delete multiple objects with the AWS SDKs, see [Delete multiple objects](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_DeleteObjects_section.html) in the *Amazon S3 API Reference*. For general information about using different AWS SDKs, see [Developing with Amazon S3 using the AWS SDKs](https://docs.aws.amazon.com/AmazonS3/latest/API/sdk-general-information-section.html) in the *Amazon S3 API Reference*. # Using the REST API You can use the AWS SDKs to delete multiple objects using the Multi-Object Delete API. However, if your application requires it, you can send REST requests directly. For more information, see [Delete Multiple Objects](https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html) in the *Amazon Simple Storage Service API Reference*. # Organizing, listing, and working with your objects In Amazon S3, you can use prefixes to organize your storage. A prefix is a logical grouping of the objects in a bucket. The prefix value is similar to a directory name that enables you to store similar data under the same directory in a bucket. When you programmatically upload objects, you can use prefixes to organize your data. In the Amazon S3 console, prefixes are called folders. You can view all your objects and folders in the S3 console by navigating to a bucket. You can also view information about each object, including object properties. For more information about listing and organizing your data in Amazon S3, see the following topics. **Topics** + [ # Organizing objects using prefixes ](using-prefixes.md) + [ # Listing object keys programmatically ](ListingKeysUsingAPIs.md) + [ # Organizing objects in the Amazon S3 console by using folders ](using-folders.md) + [ # Viewing object properties in the Amazon S3 console ](view-object-properties.md) + [ # Categorizing your objects using tags ](object-tagging.md) # Organizing objects using prefixes You can use prefixes to organize the data that you store in Amazon S3 buckets. A prefix is a string of characters at the beginning of the object key name. A prefix can be any length, subject to the maximum length of the object key name (1,024 bytes). You can think of prefixes as a way to organize your data in a similar way to directories. However, prefixes are not directories. Searching by prefix limits the results to only those keys that begin with the specified prefix. The delimiter causes a list operation to roll up all the keys that share a common prefix into a single summary list result. The purpose of the prefix and delimiter parameters is to help you organize and then browse your keys hierarchically. To do this, first pick a delimiter for your bucket, such as slash (/), that doesn't occur in any of your anticipated key names. You can use another character as a delimiter. There is nothing unique about the slash (/) character, but it is a very common prefix delimiter. Next, construct your key names by concatenating all containing levels of the hierarchy, separating each level with the delimiter. For example, if you were storing information about cities, you might naturally organize them by continent, then by country, then by province or state. Because these names don't usually contain punctuation, you might use slash (/) as the delimiter. The following examples use a slash (/) delimiter. + Europe/France/Nouvelle-Aquitaine/Bordeaux + North America/Canada/Quebec/Montreal + North America/USA/Washington/Bellevue + North America/USA/Washington/Seattle If you stored data for every city in the world in this manner, it would become awkward to manage a flat key namespace. By using `Prefix` and `Delimiter` with the list operation, you can use the hierarchy that you've created to list your data. For example, to list all the states in USA, set `Delimiter='/'` and `Prefix='North America/USA/'`. To list all the provinces in Canada for which you have data, set `Delimiter='/'` and `Prefix='North America/Canada/'`. For more information about delimiters, prefixes, and nested folders, see [Difference between prefixes and nested folders](https://repost.aws/knowledge-center/s3-prefix-nested-folders-difference). ## Listing objects using prefixes and delimiters If you issue a list request with a delimiter, you can browse your hierarchy at only one level, skipping over and summarizing the (possibly millions of) keys nested at deeper levels. For example, assume that you have a bucket (*amzn-s3-demo-bucket*) with the following keys: `sample.jpg` `photos/2006/January/sample.jpg` `photos/2006/February/sample2.jpg` `photos/2006/February/sample3.jpg` `photos/2006/February/sample4.jpg` The sample bucket has only the `sample.jpg` object at the root level. To list only the root level objects in the bucket, you send a GET request on the bucket with the slash (`/`) delimiter character. In response, Amazon S3 returns the `sample.jpg` object key because it does not contain the `/` delimiter character. All other keys contain the delimiter character. Amazon S3 groups these keys and returns a single `CommonPrefixes` element with the prefix value `photos/`, which is a substring from the beginning of these keys to the first occurrence of the specified delimiter. **Example** ``` 1. 2. amzn-s3-demo-bucket 3. 4. 5. 1000 6. / 7. false 8. 9. sample.jpg 10. 2011-07-24T19:39:30.000Z 11. "d1a7fb5eab1c16cb4f7cf341cf188c3d" 12. 6 13. 14. 75cc57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a 15. 16. STANDARD 17. 18. 19. photos/ 20. 21. ``` For more information about listing object keys programmatically, see [Listing object keys programmatically](ListingKeysUsingAPIs.md). # Listing object keys programmatically In Amazon S3, keys can be listed by prefix. You can choose a common prefix for the names of related keys and mark these keys with a special character that delimits hierarchy. You can then use the list operation to select and browse keys hierarchically. This is similar to how files are stored in directories within a file system. Amazon S3 exposes a list operation that lets you enumerate the keys contained in a bucket. Keys are selected for listing by bucket and prefix. For example, consider a bucket named "`dictionary`" that contains a key for every English word. You might make a call to list all the keys in that bucket that start with the letter "q". List results are always returned in UTF-8 binary order. Both the SOAP and REST list operations return an XML document that contains the names of matching keys and information about the object identified by each key. **Note** SOAP APIs for Amazon S3 are not available for new customers, and are approaching End of Life (EOL) on August 31, 2025. We recommend that you use either the REST API or the AWS SDKs. Groups of keys that share a prefix terminated by a special delimiter can be rolled up by that common prefix for the purposes of listing. This enables applications to organize and browse their keys hierarchically, much like how you would organize your files into directories in a file system. For example, to extend the dictionary bucket to contain more than just English words, you might form keys by prefixing each word with its language and a delimiter, such as "`French/logical`". Using this naming scheme and the hierarchical listing feature, you could retrieve a list of only French words. You could also browse the top-level list of available languages without having to iterate through all the lexicographically intervening keys. For more information about this aspect of listing, see [Organizing objects using prefixes](using-prefixes.md). **REST API** If your application requires it, you can send REST requests directly. You can send a GET request to return some or all of the objects in a bucket or you can use selection criteria to return a subset of the objects in a bucket. For more information, see [GET Bucket (List Objects) Version 2](https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html) in the *Amazon Simple Storage Service API Reference*. **List implementation efficiency** List performance is not substantially affected by the total number of keys in your bucket. It's also not affected by the presence or absence of the `prefix`, `marker`, `maxkeys`, or `delimiter` arguments. **Iterating through multipage results** As buckets can contain a virtually unlimited number of keys, the complete results of a list query can be extremely large. To manage large result sets, the Amazon S3 API supports pagination to split them into multiple responses. Each list keys response returns a page of up to 1,000 keys with an indicator indicating if the response is truncated. You send a series of list keys requests until you have received all the keys. AWS SDK wrapper libraries provide the same pagination. ## Examples When you list all of the objects in your bucket, note that you must have the `s3:ListBucket` permission. ------ #### [ CLI ] **list-objects** The following example uses the `list-objects` command to display the names of all the objects in the specified bucket: ``` aws s3api list-objects --bucket text-content --query 'Contents[].{Key: Key, Size: Size}' ``` The example uses the `--query` argument to filter the output of `list-objects` down to the key value and size for each object For more information about objects, see [Working with objects in Amazon S3](uploading-downloading-objects.md). + For API details, see [ListObjects](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/list-objects.html) in *AWS CLI Command Reference*. **ls** The following example lists all objects and prefixes in a bucket by using the `ls` command. To use this example command, replace **amzn-s3-demo-bucket** with the name of your bucket. ``` $ aws s3 ls s3://amzn-s3-demo-bucket ``` + For more information about the high-level command `ls`, see [List buckets and objects](https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-listing-buckets) in *AWS Command Line Interface User Guide*. ------ #### [ PowerShell ] **Tools for PowerShell V4** **Example 1: This command retrieves the information about all of the items in the bucket "test-files".** ``` Get-S3Object -BucketName amzn-s3-demo-bucket ``` **Example 2: This command retrieves the information about the item "sample.txt" from bucket "test-files".** ``` Get-S3Object -BucketName amzn-s3-demo-bucket -Key sample.txt ``` **Example 3: This command retrieves the information about all items with the prefix "sample" from bucket "test-files".** ``` Get-S3Object -BucketName amzn-s3-demo-bucket -KeyPrefix sample ``` + For API details, see [ListObjects](https://docs.aws.amazon.com/powershell/v4/reference) in *AWS Tools for PowerShell Cmdlet Reference (V4)*. **Tools for PowerShell V5** **Example 1: This command retrieves the information about all of the items in the bucket "test-files".** ``` Get-S3Object -BucketName amzn-s3-demo-bucket ``` **Example 2: This command retrieves the information about the item "sample.txt" from bucket "test-files".** ``` Get-S3Object -BucketName amzn-s3-demo-bucket -Key sample.txt ``` **Example 3: This command retrieves the information about all items with the prefix "sample" from bucket "test-files".** ``` Get-S3Object -BucketName amzn-s3-demo-bucket -KeyPrefix sample ``` + For API details, see [ListObjects](https://docs.aws.amazon.com/powershell/v5/reference) in *AWS Tools for PowerShell Cmdlet Reference (V5)*. ------ # Organizing objects in the Amazon S3 console by using folders In Amazon S3 general purpose buckets, objects are the primary resources, and objects are stored in buckets. Amazon S3 general purpose buckets have a flat structure instead of a hierarchy like you would see in a file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the *folder* concept as a means of grouping objects. The console does this by using a shared name *prefix* for the grouped objects. In other words, the grouped objects have names that begin with a common string. This common string, or shared prefix, is the folder name. Object names are also referred to as *key names*. For example, you can create a folder in a general purpose bucket in the console named `photos` and store an object named `myphoto.jpg` in it. The object is then stored with the key name `photos/myphoto.jpg`, where `photos/` is the prefix. Here are two more examples: + If you have three objects in your general purpose bucket—`logs/date1.txt`, `logs/date2.txt`, and `logs/date3.txt`—the console will show a folder named `logs`. If you open the folder in the console, you will see three objects: `date1.txt`, `date2.txt`, and `date3.txt`. + If you have an object named `photos/2017/example.jpg`, the console shows you a folder named `photos` that contains the folder `2017`. The folder `2017` contains the object `example.jpg`. You can have folders within folders, but not buckets within buckets. You can upload and copy objects directly into a folder. Folders can be created, deleted, and made public, but they can't be renamed. Objects can be copied from one folder to another. **Important** When you create a folder in Amazon S3 console, S3 creates a 0-byte object. This object key is set to the folder name that you provided plus a trailing forward slash (`/`) character. For example, in Amazon S3 console, if you create a folder named `photos` in your bucket, the Amazon S3 console creates a 0-byte object with the key `photos/`. The console creates this object to support the idea of folders. Also, any pre-existing object that's named with a trailing forward slash character (`/`) appears as a folder in the Amazon S3 console. For example, an object with the key name `examplekeyname/` appears as a folder in Amazon S3 console and not as an object. Otherwise, it behaves like any other object and can be viewed and manipulated through the AWS Command Line Interface (AWS CLI), AWS SDKs, or REST API. Additionally, you can't upload an object that has a key name with a trailing forward slash character (`/`) by using the Amazon S3 console. However, you can upload objects that are named with a trailing forward slash (`/`) character by using the AWS Command Line Interface (AWS CLI), AWS SDKs, or REST API. Moreover, the Amazon S3 console doesn't display the content and metadata for folder objects like it does for other objects. When you use the console to copy an object named with a trailing forward slash character (`/`), a new folder is created in the destination location, but the object's data and metadata aren't copied. Also, a forward slash (`/`) in object key names might require special handling. For more information, see [Naming Amazon S3 objects](object-keys.md). To create folders in directory buckets, upload a folder. For more information, see [Uploading objects to a directory bucket](directory-buckets-objects-upload.md). **Topics** + [ ## Creating a folder ](#create-folder) + [ ## Making folders public ](#public-folders) + [ ## Calculating folder size ](#calculate-folder) + [ ## Deleting folders ](#delete-folders) ## Creating a folder This section describes how to use the Amazon S3 console to create a folder. **Important** If your bucket policy prevents uploading objects to this bucket without tags, metadata, or access control list (ACL) grantees, you can't create a folder by using the following procedure. Instead, upload an empty folder and specify the following settings in the upload configuration. **To create a folder** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the buckets list, choose the name of the bucket that you want to create a folder in. 1. On the **Objects** tab, choose **Create folder**. 1. Enter a name for the folder (for example, **favorite-pics**). **Note** Folder names are subject to certain limitations and guidelines, and are considered part of an object's object key name, which is limited to 1,024 bytes. For more information, see [Naming Amazon S3 objects](object-keys.md). 1. (Optional) If your bucket policy requires objects to be encrypted with a specific encryption key, under **Server-side encryption**, you must choose **Specify an encryption key** and specify the same encryption key when you create a folder. Otherwise, folder creation will fail. 1. Choose **Create folder**. ## Making folders public We recommend blocking all public access to your Amazon S3 folders and buckets unless you specifically require a public folder or bucket. When you make a folder public, anyone on the internet can view all the objects that are grouped in that folder. In the Amazon S3 console, you can make a folder public. You can also make a folder public by creating a bucket policy that limits data access by prefix. For more information, see [Identity and Access Management for Amazon S3](security-iam.md). **Warning** After you make a folder public in the Amazon S3 console, you can't make it private again. Instead, you must set permissions on each individual object in the public folder so that the objects have no public access. For more information, see [Configuring ACLs](managing-acls.md). **Topics** + [ ## Creating a folder ](#create-folder) + [ ## Making folders public ](#public-folders) + [ ## Calculating folder size ](#calculate-folder) + [ ## Deleting folders ](#delete-folders) ## Calculating folder size This section describes how to use the Amazon S3 console to calculate a folder's size. **To calculate a folder's size** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the **General purpose buckets** list, choose the name of the bucket in which your folder is stored. 1. In the **Objects** list, select the checkbox next to the name of the folder. 1. Choose **Actions**, and then choose **Calculate total size**. **Note** When you navigate away from the page, the folder information (including the total size) is no longer available. You must calculate the total size again if you want to see it again. **Important** When you use the **Calculate total size** action on specified objects or folders within your bucket, Amazon S3 calculates the total number of objects and the total storage size. However, incomplete or in-progress multipart uploads and previous or noncurrent versions aren't calculated in the total number of objects or the total size. This action calculates only the total number of objects and the total size for the current or newest version of each object that's stored in the bucket. For example, if there are two versions of an object in your bucket, then the storage calculator in Amazon S3 counts them as only one object. As a result, the total number of objects that's calculated in the Amazon S3 console can differ from the **Object Count** metric shown in S3 Storage Lens and from the number reported by the Amazon CloudWatch metric, `NumberOfObjects`. Likewise, the total storage size can also differ from the **Total Storage** metric shown in S3 Storage Lens and from the `BucketSizeBytes` metric shown in CloudWatch. ## Deleting folders This section explains how to use the Amazon S3 console to delete folders from an S3 bucket. For information about Amazon S3 features and pricing, see [Amazon S3](https://aws.amazon.com/s3/). **To delete folders from an S3 bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the **General purpose buckets** list, choose the name of the bucket that you want to delete folders from. 1. In the **Objects** list, select the checkboxes next to the folders and objects that you want to delete. 1. Choose **Delete**. 1. On the **Delete objects** page, verify that the names of the folders and objects that you selected for deletion are listed under **Specified objects**. 1. In the **Delete objects** box, enter **delete**, and choose **Delete objects**. **Warning** This action deletes all specified objects. When deleting folders, wait for the delete action to finish before adding new objects to the folder. Otherwise, new objects might be deleted as well. # Viewing object properties in the Amazon S3 console You can use the Amazon S3 console to view the properties of an object, including storage class, encryption settings, tags, and metadata. **To view the properties of an object** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets** or **Directory buckets**. 1. In the bucket list, choose the name of the bucket that contains the object. 1. In the **Objects** list, choose the name of the object you want to view properties for. The **Object overview** for your object opens. You can scroll down to view the object properties. 1. On the **Object overview** page, you can view or configure the following properties for the object. **Note** If you change the **Storage Class**, ** Encryption**, or **Metadata** properties, a new object is created to replace the old one. If S3 Versioning is enabled, a new version of the object is created, and the existing object becomes an older version. The role that changes the property also becomes the owner of the new object or (object version). If you change the **Storage Class**, **Encryption**, or **Metadata** properties for an object that has user-defined tags, you must have the `s3:GetObjectTagging` permission. If you're changing these properties for an object that doesn't have user-defined tags but is over 16 MB in size, you must also have the `s3:GetObjectTagging` permission. If the destination bucket policy denies the `s3:GetObjectTagging` action, these properties for the object will be updated, but the user-defined tags will be removed from the object, and you will receive an error. 1. **Storage class** – Each object in Amazon S3 has a storage class associated with it. The storage class that you choose to use depends on how frequently you access the object. The default storage class for S3 objects in general purpose buckets is STANDARD. The default storage class for S3 objects in directory buckets is S3 Express One Zone. You choose which storage class to use when you upload an object. For more information about storage classes, see [Understanding and managing Amazon S3 storage classes](storage-class-intro.md). To change the storage class after you upload an object to a general purpose bucket, choose **Storage class**. Choose the storage class that you want, and then choose **Save**. **Note** Storage class of objects in a directory bucket cannot be changed. 1. **Server-side encryption settings** – You can use server-side encryption to encrypt your S3 objects. For more information, see [Specifying server-side encryption with AWS KMS (SSE-KMS)](specifying-kms-encryption.md) or [Specifying server-side encryption with Amazon S3 managed keys (SSE-S3)](specifying-s3-encryption.md). 1. **Metadata** – Each object in Amazon S3 has a set of name-value pairs that represents its metadata. For information about adding metadata to an S3 object, see [Editing object metadata in the Amazon S3 console](add-object-metadata.md). 1. **Tags** – You categorize storage by adding tags to an S3 object in a general purpose bucket. For more information, see [Categorizing your objects using tags](object-tagging.md). 1. **Object lock legal hold and retention** – You can prevent an object in a general purpose bucket from being deleted. For more information, see [Locking objects with Object Lock](object-lock.md). # Categorizing your objects using tags Use object tagging to categorize storage. Each tag is a key-value pair. You can add tags to new objects when you upload them, or you can add them to existing objects. + You can associate up to 10 tags with an object. Tags that are associated with an object must have unique tag keys. + A tag key can be up to 128 Unicode characters in length, and tag values can be up to 256 Unicode characters in length. Amazon S3 object tags are internally represented in UTF-16. Note that in UTF-16, characters consume either 1 or 2 character positions. + The key and values are case sensitive. + For more information about tag restrictions, see [User-defined tag restrictions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/custom-tags.html#allocation-tag-restrictions) in the *AWS Billing and Cost Management User Guide*. For basic tag restrictions, see [Tag restrictions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions) in the *Amazon EC2 User Guide*. **Examples** Consider the following tagging examples: **Example PHI information** Suppose that an object contains protected health information (PHI) data. You might tag the object using the following key-value pair. ``` PHI=True ``` or ``` Classification=PHI ``` **Example Project files** Suppose that you store project files in your S3 bucket. You might tag these objects with a key named `Project` and a value, as shown following. ``` Project=Blue ``` **Example Multiple tags** You can add multiple tags to an object, as shown following. ``` Project=x Classification=confidential ``` **Key name prefixes and tags** Object key name prefixes also enable you to categorize storage. However, prefix-based categorization is one-dimensional. Consider the following object key names: ``` photos/photo1.jpg project/projectx/document.pdf project/projecty/document2.pdf ``` These key names have the prefixes `photos/`, `project/projectx/`, and `project/projecty/`. These prefixes enable one-dimensional categorization. That is, everything under a prefix is one category. For example, the prefix `project/projectx` identifies all documents related to project x. With tagging, you now have another dimension. If you want photo1 in project x category, you can tag the object accordingly. **Additional benefits** In addition to data classification, tagging offers benefits such as the following: + Object tags enable fine-grained access control of permissions. For example, you could grant a user permissions to read-only objects with specific tags. + Object tags enable fine-grained object lifecycle management in which you can specify a tag-based filter, in addition to a key name prefix, in a lifecycle rule. + When using Amazon S3 analytics, you can configure filters to group objects together for analysis by object tags, by key name prefix, or by both prefix and tags. + You can also customize Amazon CloudWatch metrics to display information by specific tag filters. The following sections provide details. **Important** It is acceptable to use tags to label objects containing confidential data, such as personally identifiable information (PII) or protected health information (PHI). However, the tags themselves shouldn't contain any confidential information. **Adding object tag sets to multiple Amazon S3 object with a single request** To add object tag sets to more than one Amazon S3 object with a single request, you can use S3 Batch Operations. You provide S3 Batch Operations with a list of objects to operate on. S3 Batch Operations calls the respective API operation to perform the specified operation. A single Batch Operations job can perform the specified operation on billions of objects containing exabytes of data. The S3 Batch Operations feature tracks progress, sends notifications, and stores a detailed completion report of all actions, providing a fully managed, auditable, serverless experience. You can use S3 Batch Operations through the Amazon S3 console, AWS CLI, AWS SDKs, or REST API. For more information, see [S3 Batch Operations basics](batch-ops.md#batch-ops-basics). For more information about object tags, see [Managing object tags](tagging-managing.md). ## API operations related to object tagging Amazon S3 supports the following API operations that are specifically for object tagging: **Object API operations** + [PUT Object tagging](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUTtagging.html) – Replaces tags on an object. You specify tags in the request body. There are two distinct scenarios of object tag management using this API. + Object has no tags – Using this API you can add a set of tags to an object (the object has no prior tags). + Object has a set of existing tags – To modify the existing tag set, you must first retrieve the existing tag set, modify it on the client side, and then use this API to replace the tag set. **Note** If you send this request with an empty tag set, Amazon S3 deletes the existing tag set on the object. If you use this method, you will be charged for a Tier 1 Request (PUT). For more information, see [Amazon S3 Pricing](https://d0.awsstatic.com/whitepapers/aws_pricing_overview.pdf). The [DELETE Object tagging](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETEtagging.html) request is preferred because it achieves the same result without incurring charges. + [GET Object tagging](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGETtagging.html) – Returns the tag set associated with an object. Amazon S3 returns object tags in the response body. + [DELETE Object tagging](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETEtagging.html) – Deletes the tag set associated with an object. **Other API operations that support tagging** + [PUT Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html) and [Initiate Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html)– You can specify tags when you create objects. You specify tags using the `x-amz-tagging` request header. + [GET Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html) – Instead of returning the tag set, Amazon S3 returns the object tag count in the `x-amz-tag-count` header (only if the requester has permissions to read tags) because the header response size is limited to 8 K bytes. If you want to view the tags, you make another request for the [GET Object tagging](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGETtagging.html) API operation. + [POST Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html) – You can specify tags in your POST request. As long as the tags in your request don't exceed the 8 K byte HTTP request header size limit, you can use the `PUT Object `API to create objects with tags. If the tags you specify exceed the header size limit, you can use this POST method in which you include the tags in the body. [PUT Object - Copy](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html) – You can specify the `x-amz-tagging-directive` in your request to direct Amazon S3 to either copy (default behavior) the tags or replace tags by a new set of tags provided in the request. Note the following: + S3 Object Tagging is strongly consistent. For more information, see [Amazon S3 data consistency model](Welcome.md#ConsistencyModel). ## Additional configurations This section explains how object tagging relates to other configurations. ### Object tagging and lifecycle management In bucket lifecycle configuration, you can specify a filter to select a subset of objects to which the rule applies. You can specify a filter based on the key name prefixes, object tags, or both. Suppose that you store photos (raw and the finished format) in your Amazon S3 bucket. You might tag these objects as shown following. ``` phototype=raw or phototype=finished ``` You might consider archiving the raw photos to Amazon Glacier sometime after they are created. You can configure a lifecycle rule with a filter that identifies the subset of objects with the key name prefix (`photos/`) that have a specific tag (`phototype=raw`). For more information, see [Managing the lifecycle of objects](object-lifecycle-mgmt.md). ### Object tagging and replication If you configured Replication on your bucket, Amazon S3 replicates tags, provided you grant Amazon S3 permission to read the tags. For more information, see [Setting up live replication overview](replication-how-setup.md). ### Object tagging event notifications You can set up an Amazon S3 event notification to receive notice when an object tag is added or deleted from an object. The `s3:ObjectTagging:Put` event type notifies you when a tag is PUT on an object or when an existing tag is updated. The `s3:ObjectTagging:Delete` event type notifies you when a tag is removed from an object. For more information, see [ Enabling event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-enable-disable-notification-intro.html). For more information about object tagging, see the following topics: **Topics** + [ ## API operations related to object tagging ](#tagging-apis) + [ ## Additional configurations ](#tagging-other-configs) + [ # Tagging and access control policies ](tagging-and-policies.md) + [ # Managing object tags ](tagging-managing.md) # Tagging and access control policies You can also use permissions policies (bucket and user policies) to manage permissions related to object tagging. For policy actions see the following topics: + [Object operations](security_iam_service-with-iam.md#using-with-s3-actions-related-to-objects) + [Bucket operations](security_iam_service-with-iam.md#using-with-s3-actions-related-to-buckets) Object tags enable fine-grained access control for managing permissions. You can grant conditional permissions based on object tags. Amazon S3 supports the following condition keys that you can use to grant conditional permissions based on object tags: + `s3:ExistingObjectTag/` – Use this condition key to verify that an existing object tag has the specific tag key and value. **Note** When granting permissions for the `PUT Object` and `DELETE Object` operations, this condition key is not supported. That is, you cannot create a policy to grant or deny a user permissions to delete or overwrite an object based on its existing tags. + `s3:RequestObjectTagKeys` – Use this condition key to restrict the tag keys that you want to allow on objects. This is useful when adding tags to objects using the PutObjectTagging and PutObject, and POST object requests. + `s3:RequestObjectTag/` – Use this condition key to restrict the tag keys and values that you want to allow on objects. This is useful when adding tags to objects using the PutObjectTagging and PutObject, and POST Bucket requests. For a complete list of Amazon S3 service-specific condition keys, see [Bucket policy examples using condition keys](amazon-s3-policy-keys.md). The following permissions policies illustrate how object tagging enables fine grained access permissions management. **Example 1: Allow a user to read only the objects that have a specific tag and key value** The following permissions policy limits a user to only reading objects that have the `environment: production` tag key and value. This policy uses the `s3:ExistingObjectTag` condition key to specify the tag key and value. **** ``` { "Version":"2012-10-17", "Statement": [ { "Principal": { "AWS": [ "arn:aws:iam::111122223333:role/JohnDoe" ] }, "Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectVersion"], "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*", "Condition": { "StringEquals": {"s3:ExistingObjectTag/environment": "production"} } } ] } ``` **Example 2: Restrict which object tag keys that users can add** The following permissions policy grants a user permissions to perform the `s3:PutObjectTagging` action, which allows user to add tags to an existing object. The condition uses the `s3:RequestObjectTagKeys` condition key to specify the allowed tag keys, such as `Owner` or `CreationDate`. For more information, see [Creating a condition that tests multiple key values](https://docs.aws.amazon.com//IAM/latest/UserGuide/reference_policies_multi-value-conditions.html) in the *IAM User Guide*. The policy ensures that every tag key specified in the request is an authorized tag key. The `ForAnyValue` qualifier in the condition ensures that at least one of the specified keys must be present in the request. **** ``` { "Version":"2012-10-17", "Statement": [ {"Principal":{"AWS":[ "arn:aws:iam::111122223333:role/JohnDoe" ] }, "Effect": "Allow", "Action": [ "s3:PutObjectTagging" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket/*" ], "Condition": {"ForAnyValue:StringEquals": {"s3:RequestObjectTagKeys": [ "Owner", "CreationDate" ] } } } ] } ``` **Example 3: Require a specific tag key and value when allowing users to add object tags** The following example policy grants a user permission to perform the `s3:PutObjectTagging` action, which allows a user to add tags to an existing object. The condition requires the user to include a specific tag key (such as `Project`) with the value set to `X`. **** ``` { "Version":"2012-10-17", "Statement": [ {"Principal":{"AWS":[ "arn:aws:iam::111122223333:user/JohnDoe" ] }, "Effect": "Allow", "Action": [ "s3:PutObjectTagging" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket/*" ], "Condition": {"StringEquals": {"s3:RequestObjectTag/Project": "X" } } } ] } ``` # Managing object tags This section explains how you can manage object tags using the AWS SDKs for Java and .NET or the Amazon S3 console. Object tagging gives you a way to categorize storage in general purpose buckets. Each tag is a key-value pair that adheres to the following rules: + You can associate up to 10 tags with an object. Tags that are associated with an object must have unique tag keys. + A tag key can be up to 128 Unicode characters in length, and tag values can be up to 256 Unicode characters in length. Amazon S3 object tags are internally represented in UTF-16. Note that in UTF-16, characters consume either 1 or 2 character positions. + The key and values are case sensitive. For more information about object tags, see [Categorizing your objects using tags](object-tagging.md). For more information about tag restrictions, see [User-Defined Tag Restrictions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html) in the *AWS Billing and Cost Management User Guide*. ## Using the S3 console **To add tags to an object** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the bucket list, choose the name of the bucket that contains the object. 1. Select the check box to the left of the names of the objects you want to change. 1. In the **Actions** menu, choose **Edit tags**. 1. Review the objects listed, and choose **Add tags**. 1. Each object tag is a key-value pair. Enter a **Key** and a **Value**. To add another tag, choose **Add Tag**. You can enter up to 10 tags for an object. 1. Choose **Save changes**. Amazon S3 adds the tags to the specified objects. For more information, see also [Viewing object properties in the Amazon S3 console](view-object-properties.md) and [Uploading objects](upload-objects.md) in this guide. ## Using the AWS SDKs ------ #### [ Java ] To manage object tags using the AWS SDK for Java, you can set tags for a new object and retrieve or replace tags for an existing object. For more information about object tagging, see [Categorizing your objects using tags](object-tagging.md). Upload an object to a bucket and set tags using an S3Client. For examples, see [Upload an object to a bucket](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_PutObject_section.html) in the *Amazon S3 API Reference*. ------ #### [ .NET ] The following example shows how to use the AWS SDK for .NET to set the tags for a new object and retrieve or replace the tags for an existing object. For more information about object tagging, see [Categorizing your objects using tags](object-tagging.md). For information about setting up and running the code examples, see [Getting Started with the AWS SDK for .NET](https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-setup.html) in the *AWS SDK for .NET Developer Guide*. ``` using Amazon; using Amazon.S3; using Amazon.S3.Model; using System; using System.Collections.Generic; using System.Threading.Tasks; namespace Amazon.DocSamples.S3 { public class ObjectTagsTest { private const string bucketName = "*** bucket name ***"; private const string keyName = "*** key name for the new object ***"; private const string filePath = @"*** file path ***"; // Specify your bucket region (an example region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 client; public static void Main() { client = new AmazonS3Client(bucketRegion); PutObjectWithTagsTestAsync().Wait(); } static async Task PutObjectWithTagsTestAsync() { try { // 1. Put an object with tags. var putRequest = new PutObjectRequest { BucketName = bucketName, Key = keyName, FilePath = filePath, TagSet = new List{ new Tag { Key = "Keyx1", Value = "Value1"}, new Tag { Key = "Keyx2", Value = "Value2" } } }; PutObjectResponse response = await client.PutObjectAsync(putRequest); // 2. Retrieve the object's tags. GetObjectTaggingRequest getTagsRequest = new GetObjectTaggingRequest { BucketName = bucketName, Key = keyName }; GetObjectTaggingResponse objectTags = await client.GetObjectTaggingAsync(getTagsRequest); for (int i = 0; i < objectTags.Tagging.Count; i++) Console.WriteLine("Key: {0}, Value: {1}", objectTags.Tagging[i].Key, objectTags.Tagging[i].Value); // 3. Replace the tagset. Tagging newTagSet = new Tagging(); newTagSet.TagSet = new List{ new Tag { Key = "Key3", Value = "Value3"}, new Tag { Key = "Key4", Value = "Value4" } }; PutObjectTaggingRequest putObjTagsRequest = new PutObjectTaggingRequest() { BucketName = bucketName, Key = keyName, Tagging = newTagSet }; PutObjectTaggingResponse response2 = await client.PutObjectTaggingAsync(putObjTagsRequest); // 4. Retrieve the object's tags. GetObjectTaggingRequest getTagsRequest2 = new GetObjectTaggingRequest(); getTagsRequest2.BucketName = bucketName; getTagsRequest2.Key = keyName; GetObjectTaggingResponse objectTags2 = await client.GetObjectTaggingAsync(getTagsRequest2); for (int i = 0; i < objectTags2.Tagging.Count; i++) Console.WriteLine("Key: {0}, Value: {1}", objectTags2.Tagging[i].Key, objectTags2.Tagging[i].Value); } catch (AmazonS3Exception e) { Console.WriteLine( "Error encountered ***. Message:'{0}' when writing an object" , e.Message); } catch (Exception e) { Console.WriteLine( "Encountered an error. Message:'{0}' when writing an object" , e.Message); } } } } ``` ------ # Download and upload objects with presigned URLs You can use presigned URLs to grant time-limited access to objects in Amazon S3 without updating your bucket policy. A presigned URL can be entered in a browser or used by a program to download an object. The credentials used by the presigned URL are those of the AWS Identity and Access Management (IAM) principal who generated the URL. You can also use presigned URLs to allow someone to upload a specific object to your Amazon S3 bucket. This allows an upload without requiring another party to have AWS security credentials or permissions. If an object with the same key already exists in the bucket as specified in the presigned URL, Amazon S3 replaces the existing object with the uploaded object. You can use the presigned URL multiple times, up to the expiration date and time. When you create a presigned URL, you must provide your security credentials, and then specify the following: + An Amazon S3 bucket + An object key (if downloading this object will be in your Amazon S3 bucket, if uploading this is the file name to be uploaded) + An HTTP method (`GET` for downloading objects, `PUT` for uploading, `HEAD` for reading object metadata, etc) + An expiration time interval When using presigned URLs to upload objects, you can verify object integrity using checksums. While presigned URLs created with AWS Signature Version 2 only support MD5 checksums, presigned URLs created with AWS Signature Version 4 support additional checksum algorithms including CRC-64/NVME, CRC32, CRC32C, SHA-1, and SHA-256. To use these additional checksum algorithms, ensure you're using AWS Signature Version 4 and include the appropriate checksum header in your upload request. For more information about object integrity, see [Checking object integrity in Amazon S3](checking-object-integrity.md). **Topics** + [ ## Who can create a presigned URL ](#who-presigned-url) + [ ## Expiration time for presigned URLs ](#PresignedUrl-Expiration) + [ ## Limiting presigned URL capabilities ](#PresignedUrlUploadObject-LimitCapabilities) + [ ## Frequently asked questions for presigned URLs ](#PresignedUrlFAQ) + [ # Sharing objects with presigned URLs ](ShareObjectPreSignedURL.md) + [ # Uploading objects with presigned URLs ](PresignedUrlUploadObject.md) ## Who can create a presigned URL Anyone with valid security credentials can create a presigned URL. But for someone to successfully access an object, the presigned URL must be created by someone who has permission to perform the operation that the presigned URL is based upon. The following are the types of credentials that you can use to create a presigned URL: + **IAM user** – Valid up to 7 days when you're using AWS Signature Version 4. To create a presigned URL that's valid for up to 7 days, first delegate IAM user credentials (the access key and secret key) to the method you're using to create the presigned URL. + **Temporary security credentials** – Can't be valid for longer than the credentials themselves. These credentials include: + **IAM role credentials** – The presigned URL expires when the role session expires, even if you specify a longer expiration time. + **IAM role credentials used by Amazon EC2 instances** – Valid for the duration of the role credentials (typically 6 hours). + **AWS Security Token Service credentials** – Valid only for the duration of the temporary credentials. **Note** If you created a presigned URL using a temporary credential, the URL expires when the credential expires. In general, a presigned URL expires when the credential you used to create it is revoked, deleted, or deactivated. This is true even if the URL was created with a later expiration time. For temporary security credentials lifetimes, see [Comparing AWS STS API operations](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) in the *IAM User Guide*. ## Expiration time for presigned URLs A presigned URL remains valid for the period of time specified when the URL is generated. If you create a presigned URL with the Amazon S3 console, the expiration time can be set between 1 minute and 12 hours. If you use the AWS CLI or AWS SDKs, the expiration time can be set as high as 7 days. If you created a presigned URL by using a temporary token, then the URL expires when the token expires. In general, a presigned URL expires when the credential you used to create it is revoked, deleted, or deactivated. This is true even if the URL was created with a later expiration time. For more information about how the credentials you use affect the expiration time, see [Who can create a presigned URL](#who-presigned-url). Amazon S3 checks the expiration date and time of a signed URL at the time of the HTTP request. For example, if a client begins to download a large file immediately before the expiration time, the download continues even if the expiration time passes during the download. However, if the connection drops and the client tries to restart the download after the expiration time passes, the download fails. ## Limiting presigned URL capabilities The capabilities of a presigned URL are limited by the permissions of the user who created it. In essence, presigned URLs are bearer tokens that grant access to those who possess them. As such, we recommend that you protect them appropriately. The following are some methods that you can use to restrict the use of your presigned URLs. **AWS Signature Version 4 (SigV4)** To enforce specific behavior when presigned URL requests are authenticated by using AWS Signature Version 4 (SigV4), you can use condition keys in bucket policies and access point policies. For example, the following bucket policy uses the `s3:signatureAge` condition to deny any Amazon S3 presigned URL request on objects in the *amzn-s3-demo-bucket* bucket if the signature is more than 10 minutes old. To use this example, replace the *`user input placeholders`* with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "Deny a presigned URL request if the signature is more than 10 min old", "Effect": "Deny", "Principal": { "AWS": "*" }, "Action": "s3:*", "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*", "Condition": { "NumericGreaterThan": { "s3:signatureAge": "600000" } } } ] } ``` ------ For more information about policy keys related AWS Signature Version 4, see [AWS Signature Version 4 Authentication](https://docs.aws.amazon.com/AmazonS3/latest/API/bucket-policy-s3-sigv4-conditions.html) in the *Amazon Simple Storage Service API Reference*. **Network path restriction** If you want to restrict the use of presigned URLs and all Amazon S3 access to particular network paths, you can write AWS Identity and Access Management (IAM) policies. You can set these policies on the IAM principal that makes the call, the Amazon S3 bucket, or both. A network-path restriction on the IAM principal requires the user of those credentials to make requests from the specified network. A restriction on the bucket or access point requires that all requests to that resource originate from the specified network. These restrictions also apply outside of the presigned URL scenario. The IAM global condition key that you use depends on the type of endpoint. If you're using the public endpoint for Amazon S3, use `aws:SourceIp`. If you're using a virtual private cloud (VPC) endpoint to Amazon S3, use `aws:SourceVpc` or `aws:SourceVpce`. The following IAM policy statement requires the principal to access AWS only from the specified network range. With this policy statement, all access must originate from that range. This includes the case of someone who's using a presigned URL for Amazon S3. To use this example, replace the *`user input placeholders`* with your own information. ``` { "Sid": "NetworkRestrictionForIAMPrincipal", "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "NotIpAddressIfExists": {"aws:SourceIp": "IP-address-range"}, "BoolIfExists": {"aws:ViaAWSService": "false"} } } ``` ## Frequently asked questions for presigned URLs **Q: Why do my presigned URLs expire earlier than the configured expiration time?** Presigned URLs remain valid only while their underlying credentials are valid. A presigned URL expires at either its configured expiration time or when its associated credentials expire, whichever occurs first. For Amazon Elastic Container Service tasks or containers, role credentials typically rotate every 1-6 hours. When using AWS Security Token Service (AWS STS) AssumeRole, the presigned URL expires when the role session ends, which by default is 1 hour. For Amazon EC2 instance profiles, metadata credentials rotate periodically with a maximum validity period of approximately 6 hours. **Q: Why am I getting a 403 Forbidden error when accessing a presigned URL?** Before generating a presigned URL, verify that you have the correct permissions configured. The IAM user or role generating the URL must have the required permissions, such as `s3:GetObject`, for the specific operation. Additionally, check that the Amazon S3 bucket policy doesn't explicitly deny access to the object. **Q: I'm getting `SignatureDoesNotMatch` errors. How do I fix this?** If you encounter `SignatureDoesNotMatch` errors when using Amazon S3 presigned URLs, consider several common causes. First, ensure that your system clock is synchronized with a Network Time Protocol (NTP) server, as even small time drifts can invalidate signatures. Next, be aware that some corporate proxies might modify headers or query strings, potentially causing signature mismatches. To troubleshoot, try testing without the proxy. Finally, verify that all request parameters—including the HTTP method, headers, and query string—match exactly between URL generation and usage. Addressing these issues can often resolve `SignatureDoesNotMatch` errors. **Q: I'm getting `ExpiredToken` errors. What should I do?** When you receive `ExpiredToken` errors while using presigned URLs, it indicates that the AWS credentials used to generate the URL are no longer valid. To resolve this issue, refresh your AWS credentials before generating new presigned URLs. For long-running applications, we recommend implementing credential refresh logic to maintain continuous access. Where appropriate, you can use longer-lived credentials or implement token refresh mechanisms. If you're using AWS Security Token Service (AWS STS) AssumeRole, verify that your configured session duration meets your use case requirements. Remember that presigned URLs remain valid only for the duration of their underlying credentials, so implementing proper credential management is essential. # Sharing objects with presigned URLs By default, all Amazon S3 objects are private, only the object owner has permission to access them. However, the object owner may share objects with others by creating a presigned URL. A presigned URL uses security credentials to grant time-limited permission to download objects. The URL can be entered in a browser or used by a program to download the object. The credentials used by the presigned URL are those of the AWS user who generated the URL. For general information about presigned URLs, see [Download and upload objects with presigned URLs](using-presigned-url.md). You can create a presigned URL for sharing an object without writing any code by using the Amazon S3 console, AWS Explorer for Visual Studio (Windows), or AWS Toolkit for Visual Studio Code. You can also generate a presigned URL programmatically by using the AWS Command Line Interface (AWS CLI) or the AWS SDKs. ## Using the S3 console You can use the Amazon S3 console to generate a presigned URL for sharing an object up to 5 TB by following these steps. When using the console the maximum expiration time for a presigned URL is 12 hours from the time of creation. **To generate a presigned URL by using the Amazon S3 console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **General purpose buckets**. 1. In the **General purpose buckets** list, choose the name of the general purpose bucket that contains the object that you want a presigned URL for. 1. In the **Objects** list, select the object that you want to create a presigned URL for. 1. On the **Object actions** menu, choose **Share with a presigned URL**. 1. Specify how long you want the presigned URL to be valid. 1. Choose **Create presigned URL**. 1. When a confirmation appears, the URL is automatically copied to your clipboard. You will see a button to copy the presigned URL if you need to copy it again. ## Using the AWS CLI The following example AWS CLI command generates a presigned URL for sharing an object from an Amazon S3 bucket. When you use the AWS CLI, the maximum expiration time for a presigned URL is 7 days from the time of creation. To use this example, replace the *`user input placeholders`* with your own information. ``` aws s3 presign s3://amzn-s3-demo-bucket/mydoc.txt --expires-in 604800 ``` The `--expires-in` parameter specifies the expiration time in seconds. **Note** For all AWS Regions launched after March 20, 2019 you need to specify the `endpoint-url` and `AWS Region` with the request. For a list of all the Amazon S3 Regions and endpoints, see [Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) in the *AWS General Reference*. ``` aws s3 presign s3://amzn-s3-demo-bucket/mydoc.txt --expires-in 604800 --region af-south-1 --endpoint-url https://s3.af-south-1.amazonaws.com ``` For more information, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html) in the *AWS CLI Command Reference*. ## Using the AWS SDKs You can generate a presigned URL programmatically by using the AWS SDKs. ------ #### [ Python ] The following Python script generates a `GET` presigned URL for sharing an object. 1. Copy the contents of the script and save it as “*get-only-url.py*” file. To use the following examples, replace the *user input placeholders* with your own information (such as your file name). ``` import argparse import boto3 from botocore.exceptions import ClientError def generate_presigned_url(s3_client, client_method, method_parameters, expires_in): """ Generate a presigned Amazon S3 URL that can be used to perform an action. :param s3_client: A Boto3 Amazon S3 client. :param client_method: The name of the client method that the URL performs. :param method_parameters: The parameters of the specified client method. :param expires_in: The number of seconds the presigned URL is valid for. :return: The presigned URL. """ try: url = s3_client.generate_presigned_url( ClientMethod=client_method, Params=method_parameters, ExpiresIn=expires_in ) except ClientError: print(f"Couldn't get a presigned URL for client method '{client_method}'.") raise return url def main(): parser = argparse.ArgumentParser() parser.add_argument("bucket", help="The name of the bucket.") parser.add_argument( "key", help="The key (path and filename) in the S3 bucket.", ) args = parser.parse_args() # By default, this will use credentials from ~/.aws/credentials s3_client = boto3.client("s3") # The presigned URL is specified to expire in 1000 seconds url = generate_presigned_url( s3_client, "get_object", {"Bucket": args.bucket, "Key": args.key}, 1000 ) print(f"Generated GET presigned URL: {url}") if __name__ == "__main__": main() ``` 1. To generate a `GET` presigned URL for sharing a file, run the following script with your bucket name and desired object path. The following command uses example values. Replace the *user input placeholders* with your own information. ``` python get-only-url.py amzn-s3-demo-bucket ``` The script will output a `GET` presigned URL: ``` Generated GET presigned URL: https://amzn-s3-demo-bucket.s3.amazonaws.com/object.txt?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Signature=vjbyNxybdZaMmLa%2ByT372YEAiv4%3D&Expires=1741978496 ``` 1. You can download the file using the generated presigned URL with curl: ``` curl -X GET "generated-presigned-url" -o "path/to/save/file" ``` For more examples of using the AWS SDKs to generate a presigned URL for sharing an object, see [Create a presigned URL for Amazon S3 by using an AWS SDK](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_Scenario_PresignedUrl_section.html). **Note** For all AWS Regions launched after March 20, 2019 you need to specify the `endpoint-url` and `AWS Region` with the request. For a list of all the Amazon S3 Regions and endpoints, see [Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) in the *AWS General Reference*. **Note** When using the AWS SDKs, the Tagging attribute must be a header and not a query parameter. All other attributes can be passed as a parameter for the presigned URL. ------ ## Using the AWS Toolkit for Visual Studio (Windows) **Note** At this time, the AWS Toolkit for Visual Studio does not support Visual Studio for Mac. 1. Install the AWS Toolkit for Visual Studio using the following instructions, [Installing and setting up the Toolkit for Visual Studio](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/setup.html) in the *AWS Toolkit for Visual Studio User Guide*. 1. Connect to AWS using the following steps, [Connecting to AWS](https://docs.aws.amazon.com/AWSToolkitVS/latest/UserGuide/connect.html) in the *AWS Toolkit for Visual Studio User Guide*. 1. In the left side panel labeled **AWS Explorer**, double-click the bucket containing your object. 1. Right-click the object you wish to have a presigned URL generated for and select **Create Pre-Signed URL...**. 1. In the pop-up window, set the expiration date and time for your presigned URL. 1. The **Object Key**, should pre-populate based on the object you selected. 1. Choose **GET** to specify that this presigned URL will be used for downloading an object. 1. Choose the **Generate** button. 1. To copy the URL to the clipboard, choose **Copy**. 1. To use the generated presigned URL, paste the URL into any browser. ## Using AWS Toolkit for Visual Studio Code If you're using Visual Studio Code, you can generate a presigned URL to share an object without writing any code by using AWS Toolkit for Visual Studio Code. For general information, see [AWS Toolkit for Visual Studio Code](https://docs.aws.amazon.com/toolkit-for-vscode/latest/userguide/welcome.html) in the *AWS Toolkit for Visual Studio Code User Guide*. For instructions on how to install the AWS Toolkit for Visual Studio Code, see [Installing the AWS Toolkit for Visual Studio Code](https://docs.aws.amazon.com/toolkit-for-vscode/latest/userguide/setup-toolkit.html) in the *AWS Toolkit for Visual Studio Code User Guide*. 1. Connect to AWS using the following steps, [Connecting to AWS Toolkit for Visual Studio Code](https://docs.aws.amazon.com/toolkit-for-vscode/latest/userguide/connect.html) in the *AWS Toolkit for Visual Studio Code User Guide*. 1. Select the AWS logo on the left panel in Visual Studio Code. 1. Under **EXPLORER**, select **S3**. 1. Choose a bucket and file and open the context menu (right-click). 1. Choose **Generate presigned URL**, and then set the expiration time (in minutes). 1. Press Enter, and the presigned URL will be copied to your clipboard. # Uploading objects with presigned URLs You may use presigned URLs to allow someone to upload an object to your Amazon S3 bucket. Using a presigned URL will allow an upload without requiring another party to have AWS security credentials or permissions. A presigned URL is limited by the permissions of the user who creates it. That is, if you receive a presigned URL to upload an object, you can upload an object only if the creator of the URL has the necessary permissions to upload that object. When someone uses the URL to upload an object, Amazon S3 creates the object in the specified bucket. If an object with the same key that is specified in the presigned URL already exists in the bucket, Amazon S3 replaces the existing object with the uploaded object. After upload, the bucket owner will own the object. For general information about presigned URLs, see [Download and upload objects with presigned URLs](using-presigned-url.md). You can create a presigned URL for uploading an object without writing any code by using AWS Explorer for Visual Studio. You can also generate a presigned URL programmatically by using the AWS SDKs. **Note** At this time, the AWS Toolkit for Visual Studio doesn't support Visual Studio for Mac. ## Using the AWS Toolkit for Visual Studio (Windows) 1. Install the AWS Toolkit for Visual Studio using the following instructions, [Installing and setting up the Toolkit for Visual Studio](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/setup.html) in the *AWS Toolkit for Visual Studio User Guide*. 1. Connect to AWS using the following steps, [Connecting to AWS](https://docs.aws.amazon.com/AWSToolkitVS/latest/UserGuide/connect.html) in the *AWS Toolkit for Visual Studio User Guide*. 1. In the left side panel labeled **AWS Explorer**, right-click the bucket you wish to have an object uploaded to. 1. Choose **Create Pre-Signed URL...**. 1. In the pop-up window, set the expiration date and time for your presigned URL. 1. For **Object Key**, set the name of the file to be uploaded. The file you're uploading must match this name exactly. If an object with the same object key already exists in the bucket, Amazon S3 will replace the existing object with the newly uploaded object. 1. Choose **PUT** to specify that this presigned URL will be used for uploading an object. 1. Choose the **Generate** button. 1. To copy the URL to the clipboard, choose **Copy**. 1. To use this URL you can send a PUT request with the `curl` command. Include the full path to your file and the presigned URL itself. ``` curl -X PUT -T "/path/to/file" "presigned URL" ``` ## Using the AWS SDKs to generate a `PUT` presigned URL for uploading a file You can generate a presigned URL that can perform an S3 action for a limited time. **Note** If you use the AWS CLI or AWS SDKs, the expiration time for presigned URLs can be set as high as 7 days. For more information, see [Expiration time for presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html#PresignedUrl-Expiration). ------ #### [ Python ] The following Python script generates a `PUT` presigned URL for uploading an object to an S3 general purpose bucket. 1. Copy the contents of the script and save it as “*put-only-url.py*” file. To use the following examples, replace the *user input placeholders* with your own information (such as your file name). ``` import argparse import boto3 from botocore.exceptions import ClientError def generate_presigned_url(s3_client, client_method, method_parameters, expires_in): """ Generate a presigned Amazon S3 URL that can be used to perform an action. :param s3_client: A Boto3 Amazon S3 client. :param client_method: The name of the client method that the URL performs. :param method_parameters: The parameters of the specified client method. :param expires_in: The number of seconds the presigned URL is valid for. :return: The presigned URL. """ try: url = s3_client.generate_presigned_url( ClientMethod=client_method, Params=method_parameters, ExpiresIn=expires_in ) except ClientError: print(f"Couldn't get a presigned URL for client method '{client_method}'.") raise return url def main(): parser = argparse.ArgumentParser() parser.add_argument("bucket", help="The name of the bucket.") parser.add_argument( "key", help="The key (path and filename) in the S3 bucket.", ) parser.add_argument( "--region", help="The AWS region where the bucket is located.", default="us-east-1" ) parser.add_argument( "--content-type", help="The content type of the file to upload.", default="application/octet-stream" ) args = parser.parse_args() # Create S3 client with explicit region configuration s3_client = boto3.client("s3", region_name=args.region) # Optionally set signature version if needed for older S3 regions # s3_client.meta.config.signature_version = 's3v4' # The presigned URL is specified to expire in 1000 seconds url = generate_presigned_url( s3_client, "put_object", { "Bucket": args.bucket, "Key": args.key, "ContentType": args.content_type # Specify content type }, 1000 ) print(f"Generated PUT presigned URL: {url}") if __name__ == "__main__": main() ``` 1. To generate a `PUT` presigned URL for uploading a file, run the following script with your bucket name and desired object path. The following command uses example values. Replace the *user input placeholders* with your own information. ``` python put-only-url.py amzn-s3-demo-bucket --region us-east-1 --content-type application/octet-stream ``` The script will output a `PUT` presigned URL: ``` Generated PUT presigned URL: https://amzn-s3-demo-bucket.s3.amazonaws.com/object.txt?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Signature=vjbyNxybdZaMmLa%2ByT372YEAiv4%3D&Expires=1741978496 ``` 1. You can now upload the file using the generated presigned URL with curl. Make sure to include the same content type that was used when generating the URL: ``` curl -X PUT -T "path/to/your/local/file" -H "Content-Type: application/octet-stream" "generated-presigned-url" ``` If you specified a different content type when generating the URL, make sure to use that same content type in the curl command. For more examples of using the AWS SDKs to generate a presigned URL for uploading an object, see [Create a presigned URL for Amazon S3 by using an AWS SDK](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_Scenario_PresignedUrl_section.html). **Troubleshooting SignatureDoesNotMatch errors** If you encounter a `SignatureDoesNotMatch` error when using presigned URLs, check the following: Verify your system time is synchronized with a reliable time server Ensure you're using the URL exactly as generated, without any modifications Check if the URL has expired and generate a new one if needed Make sure the content type in your upload request matches the content type specified when generating the URL Confirm you're using the correct region for the bucket When using curl, enclose the URL in quotes to properly handle special characters ------ # Transforming objects with S3 Object Lambda **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). With Amazon S3 Object Lambda, you can add your own code to Amazon S3 `GET`, `LIST`, and `HEAD` requests to modify and process data as it is returned to an application. You can use custom code to modify the data returned by S3 `GET` requests to filter rows, dynamically resize and watermark images, redact confidential data, and more. You can also use S3 Object Lambda to modify the output of S3 `LIST` requests to create a custom view of all objects in a bucket and S3 `HEAD` requests to modify object metadata such as object name and size. You can use S3 Object Lambda as an origin for your Amazon CloudFront distribution to tailor data for end users, such as automatically resizing images, transcoding older formats (like from JPEG to WebP), or stripping metadata. For more information, see the AWS Blog post [Use Amazon S3 Object Lambda with Amazon CloudFront](https://aws.amazon.com/blogs/aws/new-use-amazon-s3-object-lambda-with-amazon-cloudfront-to-tailor-content-for-end-users/). Powered by AWS Lambda functions, your code runs on infrastructure that is fully managed by AWS. Using S3 Object Lambda reduces the need to create and store derivative copies of your data or to run proxies, all with no need to change your applications. **How S3 Object Lambda works** S3 Object Lambda uses AWS Lambda functions to automatically process the output of standard S3 `GET`, `LIST`, or `HEAD` requests. AWS Lambda is a serverless compute service that runs customer-defined code without requiring management of underlying compute resources. You can author and run your own custom Lambda functions, tailoring the data transformation to your specific use cases. After you configure a Lambda function, you attach it to an S3 Object Lambda service endpoint, known as an *Object Lambda Access Point*. The Object Lambda Access Point uses a standard S3 access point, known as a *supporting access point*, to access data. When you send a request to your Object Lambda Access Point, Amazon S3 automatically calls your Lambda function. Any data retrieved by using an S3 `GET`, `LIST`, or `HEAD` request through the Object Lambda Access Point returns a transformed result back to the application. All other requests are processed as normal, as illustrated in the following diagram. ![\[Diagram, showing how S3 Object Lambda works.\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/images/ObjectLamdaDiagram.png) The topics in this section describe how to work with S3 Object Lambda. **Topics** + [ # Creating Object Lambda Access Points ](olap-create.md) + [ # Using Amazon S3 Object Lambda Access Points ](olap-use.md) + [ # Security considerations for S3 Object Lambda Access Points ](olap-security.md) + [ # Writing Lambda functions for S3 Object Lambda Access Points ](olap-writing-lambda.md) + [ # Using AWS built Lambda functions ](olap-examples.md) + [ # Best practices and guidelines for S3 Object Lambda ](olap-best-practices.md) + [ # S3 Object Lambda tutorials ](olap-tutorials.md) + [ # Debugging and troubleshooting S3 Object Lambda ](olap-debugging-lambda.md) # Creating Object Lambda Access Points **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). An Object Lambda Access Point is associated with exactly one standard access point, which you specify during creation. To create an Object Lambda Access Point, you need the following resources: + **A standard S3 access point.** When you're working with Object Lambda Access Points, this standard access point is known as a *supporting access point* and is attached to an S3 bucket or Amazon FSx for OpenZFS volume. For information about creating standard access points, see [Creating an access point](creating-access-points.md). + **An AWS Lambda function.** You can either create your own Lambda function, or you can use a prebuilt function. For more information about creating Lambda functions, see [Writing Lambda functions for S3 Object Lambda Access Points](olap-writing-lambda.md). For more information about prebuilt functions, see [Using AWS built Lambda functions](olap-examples.md). + **(Optional) An AWS Identity and Access Management (IAM) policy.** Amazon S3 access points support IAM resource policies that you can use to control the use of the access point by resource, user, or other conditions. For more information about creating these policies, see [Configuring IAM policies for Object Lambda Access Points](olap-policies.md). The following sections describe how to create an Object Lambda Access Point by using: + The AWS Management Console + The AWS Command Line Interface (AWS CLI) + An AWS CloudFormation template + The AWS Cloud Development Kit (AWS CDK) For information about how to create an Object Lambda Access Point by using the REST API, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateAccessPointForObjectLambda.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateAccessPointForObjectLambda.html) in the *Amazon Simple Storage Service API Reference*. ## Create an Object Lambda Access Point Use one of the following procedures to create your Object Lambda Access Point. ### Using the S3 console **To create an Object Lambda Access Point by using the console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the navigation bar, choose the name of the currently displayed AWS Region. Next, choose the Region that you want to switch to. 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose **Create Object Lambda Access Point**. 1. For **Object Lambda Access Point name**, enter the name that you want to use for the access point. As with standard access points, there are rules for naming Object Lambda Access Points. For more information, see [Naming rules for access points](access-points-restrictions-limitations-naming-rules.md#access-points-names). 1. For **Supporting Access Point**, enter or browse to the standard access point that you want to use. The access point must be in the same AWS Region as the objects that you want to transform. For information about creating standard access points, see [Creating an access point](creating-access-points.md). 1. Under **Transformation configuration**, you can add a function that transforms your data for your Object Lambda Access Point. Do one of the following: + If you already have a AWS Lambda function in your account you can choose it under **Invoke Lambda function**. Here you may enter the Amazon Resource Name (ARN) of an Lambda function in your AWS account or choose a Lambda function from the drop-down menu. + If you wish to use a AWS built function choose the function name under **AWS built function** and select **Create Lambda function**. This will take you to the Lambda console where you can deploy a built function into your AWS account. For more information about built functions, see [Using AWS built Lambda functions](olap-examples.md). Under **S3 APIs**, choose one or more API operations to invoke. For each API selected you must specify a Lambda function to invoke. 1. (Optional) Under **Payload**, add JSON text that you want to provide to your Lambda function as input. You can configure payloads with different parameters for different Object Lambda Access Points that invoke the same Lambda function, thereby extending the flexibility of your Lambda function. **Important** When you're using Object Lambda Access Points, make sure that the payload does not contain any confidential information. 1. (Optional) For **Range and part number**, you must enable this option if you want to process `GET` and `HEAD` requests with range and part number headers. Enabling this option confirms that your Lambda function can recognize and process these requests. For more information about range headers and part numbers, see [Working with Range and partNumber headers](range-get-olap.md). 1. (Optional) For **Request metrics**, choose **Enable** or **Disable** to add Amazon S3 monitoring to your Object Lambda Access Point. Request metrics are billed at the standard Amazon CloudWatch rate. 1. (Optional) Under **Object Lambda Access Point policy**, set a resource policy. Resource policies grant permissions for the specified Object Lambda Access Point and can control the use of the access point by resource, user, or other conditions. For more information about Object Lambda Access Point resource policies see, [Configuring IAM policies for Object Lambda Access Points](olap-policies.md). 1. Under **Block Public Access settings for this Object Lambda Access Point**, select the block public access settings that you want to apply. All block public access settings are enabled by default for new Object Lambda Access Points, and we recommend that you leave default settings enabled. Amazon S3 currently doesn't support changing an Object Lambda Access Point's block public access settings after the Object Lambda Access Points has been created. For more information about using Amazon S3 Block Public Access, see [Managing public access to access points for general purpose buckets](access-points-bpa-settings.md). 1. Choose **Create Object Lambda Access Point**. ### Using the AWS CLI **To create an Object Lambda Access Point by using an AWS CloudFormation template** **Note** To use the following commands, replace the `user input placeholders` with your own information. 1. Download the AWS Lambda function deployment package `s3objectlambda_deployment_package.zip` at [S3 Object Lambda default configuration](https://github.com/aws-samples/amazon-s3-object-lambda-default-configuration). 1. Run the following `put-object` command to upload the package to an Amazon S3 bucket. ``` aws s3api put-object --bucket Amazon S3 bucket name --key s3objectlambda_deployment_package.zip --body release/s3objectlambda_deployment_package.zip ``` 1. Download the AWS CloudFormation template `s3objectlambda_defaultconfig.yaml` at [S3 Object Lambda default configuration](https://github.com/aws-samples/amazon-s3-object-lambda-default-configuration). 1. Run the following `deploy` command to deploy the template to your AWS account. ``` aws cloudformation deploy --template-file s3objectlambda_defaultconfig.yaml \ --stack-name CloudFormation stack name \ --parameter-overrides ObjectLambdaAccessPointName=Object Lambda Access Point name \ SupportingAccessPointName=Amazon S3 access point S3BucketName=Amazon S3 bucket \ LambdaFunctionS3BucketName=Amazon S3 bucket containing your Lambda package \ LambdaFunctionS3Key=Lambda object key LambdaFunctionS3ObjectVersion=Lambda object version \ LambdaFunctionRuntime=Lambda function runtime --capabilities capability_IAM ``` You can configure this AWS CloudFormation template to invoke Lambda for `GET`, `HEAD`, and `LIST` API operations. For more information about modifying the template's default configuration, see [Automate S3 Object Lambda setup with a CloudFormation template](olap-using-cfn-template.md). **To create an Object Lambda Access Point by using the AWS CLI** **Note** To use the following commands, replace the `user input placeholders` with your own information. The following example creates an Object Lambda Access Point named *`my-object-lambda-ap`* for the bucket *`amzn-s3-demo-bucket1`* in the account *`111122223333`*. This example assumes that a standard access point named *`example-ap`* has already been created. For information about creating a standard access point, see [Creating an access point](creating-access-points.md). This example uses the AWS prebuilt function `decompress`. For more information about prebuilt functions, see [Using AWS built Lambda functions](olap-examples.md). 1. Create a bucket. In this example, we will use *`amzn-s3-demo-bucket1`*. For information about creating buckets, see [Creating a general purpose bucket](create-bucket-overview.md). 1. Create a standard access point and attach it to your bucket. In this example, we will use *`example-ap`*. For information about creating standard access points, see [Creating an access point](creating-access-points.md). 1. Do one of the following: + Create a Lambda function in your account that you would like to use to transform your Amazon S3 object. For more information about creating Lambda functions, see [Writing Lambda functions for S3 Object Lambda Access Points](olap-writing-lambda.md). To use your custom function with the AWS CLI, see [Using Lambda with the AWS CLI](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-awscli.html) in the *AWS Lambda Developer Guide*. + Use an AWS prebuilt Lambda function. For more information about prebuilt functions, see [Using AWS built Lambda functions](olap-examples.md). 1. Create a JSON configuration file named `my-olap-configuration.json`. In this configuration, provide the supporting access point and the Amazon Resource Name (ARN) for the Lambda function that you created in the previous steps or the ARN for the prebuilt function that you're using. **Example** ``` { "SupportingAccessPoint" : "arn:aws:s3:us-east-1:111122223333:accesspoint/example-ap", "TransformationConfigurations": [{ "Actions" : ["GetObject", "HeadObject", "ListObjects", "ListObjectsV2"], "ContentTransformation" : { "AwsLambda": { "FunctionPayload" : "{\"compressionType\":\"gzip\"}", "FunctionArn" : "arn:aws:lambda:us-east-1:111122223333:function/compress" } } }] } ``` 1. Run the `create-access-point-for-object-lambda` command to create your Object Lambda Access Point. ``` aws s3control create-access-point-for-object-lambda --account-id 111122223333 --name my-object-lambda-ap --configuration file://my-olap-configuration.json ``` 1. (Optional) Create a JSON policy file named `my-olap-policy.json`. Adding an Object Lambda Access Point resource policy can control the use of the access point by resource, user, or other conditions. This resource policy grants the `GetObject` permission for account *`444455556666`* to the specified Object Lambda Access Point. **Example** ``` { "Version": "2008-10-17", "Statement": [ { "Sid": "Grant account 444455556666 GetObject access", "Effect": "Allow", "Action": "s3-object-lambda:GetObject", "Principal": { "AWS": "arn:aws:iam::444455556666:root" }, "Resource": "your-object-lambda-access-point-arn" } ] } ``` 1. (Optional) Run the `put-access-point-policy-for-object-lambda` command to set your resource policy. ``` aws s3control put-access-point-policy-for-object-lambda --account-id 111122223333 --name my-object-lambda-ap --policy file://my-olap-policy.json ``` 1. (Optional) Specify a payload. A payload is optional JSON that you can provide to your AWS Lambda function as input. You can configure payloads with different parameters for different Object Lambda Access Points that invoke the same Lambda function, thereby extending the flexibility of your Lambda function. The following Object Lambda Access Point configuration shows a payload with two parameters. ``` { "SupportingAccessPoint": "AccessPointArn", "CloudWatchMetricsEnabled": false, "TransformationConfigurations": [{ "Actions": ["GetObject", "HeadObject", "ListObjects", "ListObjectsV2"], "ContentTransformation": { "AwsLambda": { "FunctionArn": "FunctionArn", "FunctionPayload": "{\"res-x\": \"100\",\"res-y\": \"100\"}" } } }] } ``` The following Object Lambda Access Point configuration shows a payload with one parameter, and with `GetObject-Range`, `GetObject-PartNumber`, `HeadObject-Range`, and `HeadObject-PartNumber` enabled. ``` { "SupportingAccessPoint":"AccessPointArn", "CloudWatchMetricsEnabled": false, "AllowedFeatures": ["GetObject-Range", "GetObject-PartNumber", "HeadObject-Range", "HeadObject-PartNumber"], "TransformationConfigurations": [{ "Action": ["GetObject", "HeadObject", "ListObjects", "ListObjectsV2"], "ContentTransformation": { "AwsLambda": { "FunctionArn":"FunctionArn", "FunctionPayload": "{\"compression-amount\": \"5\"}" } } }] } ``` **Important** When you're using Object Lambda Access Points, make sure that the payload does not contain any confidential information. ### Using the AWS CloudFormation console and template You can create an Object Lambda Access Point by using the default configuration provided by Amazon S3. You can download an AWS CloudFormation template and Lambda function source code from the [GitHub repository](https://github.com/aws-samples/amazon-s3-object-lambda-default-configuration) and deploy these resources to set up a functional Object Lambda Access Point. For information about modifying the AWS CloudFormation template's default configuration, see [Automate S3 Object Lambda setup with a CloudFormation template](olap-using-cfn-template.md). For information about configuring Object Lambda Access Points by using CloudFormation without the template, see [https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-s3objectlambda-accesspoint.html](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-s3objectlambda-accesspoint.html) in the *AWS CloudFormation User Guide*. **To upload the Lambda function deployment package** 1. Download the AWS Lambda function deployment package `s3objectlambda_deployment_package.zip` at [S3 Object Lambda default configuration](https://github.com/aws-samples/amazon-s3-object-lambda-default-configuration). 1. Upload the package to an Amazon S3 bucket. **To create an Object Lambda Access Point by using the AWS CloudFormation console** 1. Download the AWS CloudFormation template `s3objectlambda_defaultconfig.yaml` at [S3 Object Lambda default configuration](https://github.com/aws-samples/amazon-s3-object-lambda-default-configuration). 1. Sign in to the AWS Management Console and open the AWS CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. Do one of the following: + If you've never used AWS CloudFormation before, on the AWS CloudFormation home page, choose **Create stack**. + If you have used AWS CloudFormation before, in the left navigation pane, choose **Stacks**. Choose **Create stack**, then choose **With new resources (standard)**. 1. For **Prerequisite - Prepare template**, choose **Template is ready**. 1. For **Specify template**, choose **Upload a template file** and upload `s3objectlambda_defaultconfig.yaml`. 1. Choose **Next**. 1. On the **Specify stack details** page, enter a name for the stack. 1. In the **Parameters** section, specify the following parameters that are defined in the stack template: 1. For **CreateNewSupportingAccessPoint**, do one of the following: + If you already have a supporting access point for the S3 bucket where you uploaded the template, choose **false**. + If you want to create a new access point for this bucket, choose **true**. 1. For **EnableCloudWatchMonitoring**, choose **true** or **false**, depending on whether you want to enable Amazon CloudWatch request metrics and alarms. 1. (Optional) For **LambdaFunctionPayload**, add JSON text that you want to provide to your Lambda function as input. You can configure payloads with different parameters for different Object Lambda Access Points that invoke the same Lambda function, thereby extending the flexibility of your Lambda function. **Important** When you're using Object Lambda Access Points, make sure that the payload does not contain any confidential information. 1. For **LambdaFunctionRuntime**, enter your preferred runtime for the Lambda function. The available choices are `nodejs14.x`, `python3.9`, `java11`. 1. For **LambdaFunctionS3BucketName**, enter the Amazon S3 bucket name where you uploaded the deployment package. 1. For **LambdaFunctionS3Key**, enter the Amazon S3 object key where you uploaded the deployment package. 1. For **LambdaFunctionS3ObjectVersion**, enter the Amazon S3 object version where you uploaded the deployment package. 1. For **ObjectLambdaAccessPointName**, enter a name for your Object Lambda Access Point. 1. For **S3BucketName**, enter the Amazon S3 bucket name that will be associated with your Object Lambda Access Point. 1. For **SupportingAccessPointName**, enter the name of your supporting access point. **Note** This is an access point that is associated with the Amazon S3 bucket that you chose in the previous step. If you do not have any access points associated with your Amazon S3 bucket, you can configure the template to create one for you by choosing **true** for **CreateNewSupportingAccessPoint**. 1. Choose **Next**. 1. On the **Configure stack options page**, choose **Next**. For more information about the optional settings on this page, see [Setting AWS CloudFormation stack options](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html) in the *AWS CloudFormation User Guide*. 1. On the **Review** page, choose **Create stack**. ### Using the AWS Cloud Development Kit (AWS CDK) For more information about configuring Object Lambda Access Points by using the AWS CDK, see [`AWS::S3ObjectLambda` Construct Library](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3objectlambda-readme.html) in the *AWS Cloud Development Kit (AWS CDK) API Reference*. # Automate S3 Object Lambda setup with a CloudFormation template **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). You can use an AWS CloudFormation template to quickly create an Amazon S3 Object Lambda Access Point. The CloudFormation template automatically creates relevant resources, configures AWS Identity and Access Management (IAM) roles, and sets up an AWS Lambda function that automatically handles requests through the Object Lambda Access Point. With the CloudFormation template, you can implement best practices, improve your security posture, and reduce errors caused by manual processes. This [GitHub repository](https://github.com/aws-samples/amazon-s3-object-lambda-default-configuration) contains the CloudFormation template and Lambda function source code. For instructions on how to use the template, see [Creating Object Lambda Access Points](olap-create.md). The Lambda function provided in the template does not run any transformation. Instead, it returns your objects as-is from the underlying data source. You can clone the function and add your own transformation code to modify and process data as it is returned to an application. For more information about modifying your function, see [Modifying the Lambda function](#modifying-lambda-function) and [Writing Lambda functions for S3 Object Lambda Access Points](olap-writing-lambda.md). ## Modifying the template **Creating a new supporting access point** S3 Object Lambda uses two access points, an Object Lambda Access Point and a standard S3 access point, which is referred to as the *supporting access point*. When you make a request to an Object Lambda Access Point, S3 either invokes Lambda on your behalf, or it delegates the request to the supporting access point, depending upon the S3 Object Lambda configuration. You can create a new supporting access point by passing the following parameter as part of the `aws cloudformation deploy` command when deploying the template. ``` CreateNewSupportingAccessPoint=true ``` **Configuring a function payload** You can configure a payload to provide supplemental data to the Lambda function by passing the following parameter as part of the `aws cloudformation deploy` command when deploying the template. ``` LambdaFunctionPayload="format=json" ``` **Enabling Amazon CloudWatch monitoring** You can enable CloudWatch monitoring by passing the following parameter as part of the `aws cloudformation deploy` command when deploying the template. ``` EnableCloudWatchMonitoring=true ``` This parameter enables your Object Lambda Access Point for Amazon S3 request metrics and creates two CloudWatch alarms to monitor client-side and server-side errors. **Note** Amazon CloudWatch usage will incur additional costs. For more information about Amazon S3 request metrics, see [Monitoring and logging access points](access-points-monitoring-logging.md). For pricing details, see [CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). **Configuring provisioned concurrency** To reduce latency, you can configure provisioned concurrency for the Lambda function that's backing the Object Lambda Access Point by editing the template to include the following lines under `Resources`. ``` LambdaFunctionVersion: Type: AWS::Lambda::Version Properties: FunctionName: !Ref LambdaFunction ProvisionedConcurrencyConfig: ProvisionedConcurrentExecutions: Integer ``` **Note** You will incur additional charges for provisioning concurrency. For more information about provisioned concurrency, see [Managing Lambda provisioned concurrency](https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html) in the *AWS Lambda Developer Guide*. For pricing details, see [AWS Lambda pricing](https://aws.amazon.com/lambda/pricing/). ## Modifying the Lambda function **Changing header values for a `GetObject` request** By default, the Lambda function forwards all headers, except `Content-Length` and `ETag`, from the presigned URL request to the `GetObject` client. Based on your transformation code in the Lambda function, you can choose to send new header values to the `GetObject` client. You can update your Lambda function to send new header values by passing them in the `WriteGetObjectResponse` API operation. For example, if your Lambda function translates text in Amazon S3 objects to a different language, you can pass a new value in the `Content-Language` header. You can do this by modifying the `writeResponse` function as follows: ``` async function writeResponse (s3Client: S3, requestContext: GetObjectContext, transformedObject: Buffer, headers: Headers): Promise> { const { algorithm, digest } = getChecksum(transformedObject); return s3Client.writeGetObjectResponse({ RequestRoute: requestContext.outputRoute, RequestToken: requestContext.outputToken, Body: transformedObject, Metadata: { 'body-checksum-algorithm': algorithm, 'body-checksum-digest': digest }, ...headers, ContentLanguage: 'my-new-language' }).promise(); } ``` For a full list of supported headers, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax](https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax) in the *Amazon Simple Storage Service API Reference*. **Returning metadata headers** You can update your Lambda function to send new header values by passing them in the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax](https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax) API operation request. ``` async function writeResponse (s3Client: S3, requestContext: GetObjectContext, transformedObject: Buffer, headers: Headers): Promise> { const { algorithm, digest } = getChecksum(transformedObject); return s3Client.writeGetObjectResponse({ RequestRoute: requestContext.outputRoute, RequestToken: requestContext.outputToken, Body: transformedObject, Metadata: { 'body-checksum-algorithm': algorithm, 'body-checksum-digest': digest, 'my-new-header': 'my-new-value' }, ...headers }).promise(); } ``` **Returning a new status code** You can return a custom status code to the `GetObject` client by passing it in the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax](https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax) API operation request. ``` async function writeResponse (s3Client: S3, requestContext: GetObjectContext, transformedObject: Buffer, headers: Headers): Promise> { const { algorithm, digest } = getChecksum(transformedObject); return s3Client.writeGetObjectResponse({ RequestRoute: requestContext.outputRoute, RequestToken: requestContext.outputToken, Body: transformedObject, Metadata: { 'body-checksum-algorithm': algorithm, 'body-checksum-digest': digest }, ...headers, StatusCode: Integer }).promise(); } ``` For a full list of supported status codes, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax](https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html#API_WriteGetObjectResponse_RequestSyntax) in the *Amazon Simple Storage Service API Reference*. **Applying `Range` and `partNumber` parameters to the source object** By default, the Object Lambda Access Point created by the CloudFormation template can handle the `Range` and `partNumber` parameters. The Lambda function applies the range or part number requested to the transformed object. To do so, the function must download the whole object and run the transformation. In some cases, your transformed object ranges might map exactly to your source object ranges. This means that requesting byte range A-B on your source object and running the transformation might produce the same result as requesting the whole object, running the transformation, and returning byte range A-B on the transformed object. In such cases, you can change the Lambda function implementation to apply the range or part number directly to the source object. This approach reduces the overall function latency and memory required. For more information, see [Working with Range and partNumber headers](range-get-olap.md). **Disabling `Range` and `partNumber` handling** By default, the Object Lambda Access Point created by the CloudFormation template can handle the `Range` and `partNumber` parameters. If you don't need this behavior, you can disable it by removing the following lines from the template: ``` AllowedFeatures: - GetObject-Range - GetObject-PartNumber - HeadObject-Range - HeadObject-PartNumber ``` **Transforming large objects** By default, the Lambda function processes the entire object in memory before it can start streaming the response to S3 Object Lambda. You can modify the function to stream the response as it performs the transformation. Doing so helps reduce the transformation latency and the Lambda function memory size. For an example implementation, see the [Stream compressed content example](olap-writing-lambda.md#olap-getobject-response). # Using Amazon S3 Object Lambda Access Points **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). Making requests through Amazon S3 Object Lambda Access Points works the same as making requests through other access points. For more information about how to make requests through an access point, see [Using Amazon S3 access points for general purpose buckets](using-access-points.md). You can make requests through Object Lambda Access Points by using the Amazon S3 console, AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. **Important** The Amazon Resource Names (ARNs) for Object Lambda Access Points use a service name of `s3-object-lambda`. Thus, Object Lambda Access Point ARNs begin with `arn:aws::s3-object-lambda`, instead of `arn:aws::s3`, which is used with other access points. ## How to find the ARN for your Object Lambda Access Point To use an Object Lambda Access Point with the AWS CLI or AWS SDKs, you need to know the Amazon Resource Name (ARN) of the Object Lambda Access Point. The following examples show how to find the ARN for an Object Lambda Access Point by using the Amazon S3 console or AWS CLI. ### Using the S3 console **To find the ARN for your Object Lambda Access Point by using the console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. Choose the option button next to the Object Lambda Access Point whose ARN you want to copy. 1. Choose **Copy ARN**. ### Using the AWS CLI **To find the ARN for your Object Lambda Access Point by using the AWS CLI** 1. To retrieve a list of the Object Lambda Access Points that are associated with your AWS account, run the following command. Before running the command, replace the account ID *`111122223333`* with your AWS account ID. ``` aws s3control list-access-points-for-object-lambda --account-id 111122223333 ``` 1. Review the command output to find the Object Lambda Access Point ARN that you want to use. The output of the previous command should look similar to the following example. ``` { "ObjectLambdaAccessPointList": [ { "Name": "my-object-lambda-ap", "ObjectLambdaAccessPointArn": "arn:aws:s3-object-lambda:us-east-1:111122223333:accesspoint/my-object-lambda-ap" }, ... ] } ``` ## How to use a bucket-style alias for your S3 bucket Object Lambda Access Point When you create an Object Lambda Access Point, Amazon S3 automatically generates a unique alias for your Object Lambda Access Point. You can use this alias instead of an Amazon S3 bucket name or the Object Lambda Access Point Amazon Resource Name (ARN) in a request for access point data plane operations. For a list of these operations, see [Access point compatibility](access-points-service-api-support.md). An Object Lambda Access Point alias name is created within the same namespace as an Amazon S3 bucket. This alias name is automatically generated and cannot be changed. For an existing Object Lambda Access Point, an alias is automatically assigned for use. An Object Lambda Access Point alias name meets all the requirements of a valid Amazon S3 bucket name and consists of the following parts: `Object Lambda Access Point name prefix-metadata--ol-s3` **Note** The `--ol-s3` suffix is reserved for Object Lambda Access Point alias names and can't be used for bucket or Object Lambda Access Point names. For more information about Amazon S3 bucket-naming rules, see [General purpose bucket naming rules](bucketnamingrules.md). The following examples show the ARN and the Object Lambda Access Point alias for an Object Lambda Access Point named `my-object-lambda-access-point`: + **ARN** – `arn:aws:s3-object-lambda:region:account-id:accesspoint/my-object-lambda-access-point` + **Object Lambda Access Point alias** – `my-object-lambda-acc-1a4n8yjrb3kda96f67zwrwiiuse1a--ol-s3` When you use an Object Lambda Access Point, you can use the Object Lambda Access Point alias name without requiring extensive code changes. When you delete an Object Lambda Access Point, the Object Lambda Access Point alias name becomes inactive and unprovisioned. ### How to find the alias for your Object Lambda Access Point #### Using the S3 console **To find the alias for your Object Lambda Access Point by using the console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. For the Object Lambda Access Point that you want to use, copy the **Object Lambda Access Point alias** value. #### Using the AWS CLI When you create an Object Lambda Access Point, Amazon S3 automatically generates an Object Lambda Access Point alias name, as shown in the following example command. To run this command, replace the `user input placeholders` with your own information. For information about how to create an Object Lambda Access Point by using the AWS CLI, see [To create an Object Lambda Access Point by using the AWS CLI](olap-create.md#olap-create-cli-specific). ``` aws s3control create-access-point-for-object-lambda --account-id 111122223333 --name my-object-lambda-access-point --configuration file://my-olap-configuration.json { "ObjectLambdaAccessPointArn": "arn:aws:s3:region:111122223333:accesspoint/my-access-point", "Alias": { "Value": "my-object-lambda-acc-1a4n8yjrb3kda96f67zwrwiiuse1a--ol-s3", "Status": "READY" } } ``` The generated Object Lambda Access Point alias name has two fields: + The `Value` field is the alias value of the Object Lambda Access Point. + The `Status` field is the status of the Object Lambda Access Point alias. If the status is `PROVISIONING`, Amazon S3 is provisioning the Object Lambda Access Point alias, and the alias is not yet ready for use. If the status is `READY`, the Object Lambda Access Point alias has been successfully provisioned and is ready for use. For more information about the `ObjectLambdaAccessPointAlias` data type in the REST API, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateAccessPointForObjectLambda.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateAccessPointForObjectLambda.html) and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_ObjectLambdaAccessPointAlias.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_ObjectLambdaAccessPointAlias.html) in the *Amazon Simple Storage Service API Reference*. ### How to use the Object Lambda Access Point alias You can use an Object Lambda Access Point alias instead of an Amazon S3 bucket name for the operations listed in [Access point compatibility](access-points-service-api-support.md). The following AWS CLI example for the `get-bucket-location` command uses the bucket's access point alias to return the AWS Region that the bucket is in. To run this command, replace the `user input placeholders` with your own information. ``` aws s3api get-bucket-location --bucket my-object-lambda-acc-w7i37nq6xuzgax3jw3oqtifiusw2a--ol-s3 { "LocationConstraint": "us-west-2" } ``` If the Object Lambda Access Point alias in a request isn't valid, the error code `InvalidAccessPointAliasError` is returned. For more information about `InvalidAccessPointAliasError`, see [List of Error Codes](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList) in the *Amazon Simple Storage Service API Reference*. The limitations of an Object Lambda Access Point alias are the same as those of an access point alias. For more information about the limitations of an access point alias, see [Access point alias limitations](access-points-naming.md#use-ap-alias-limitations). # Security considerations for S3 Object Lambda Access Points **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). With Amazon S3 Object Lambda, you can perform custom transformations on data as it leaves Amazon S3 by using the scale and flexibility of AWS Lambda as a compute platform. S3 and Lambda remain secure by default, but to maintain this security, special consideration by the Lambda function author is required. S3 Object Lambda requires that all access be made by authenticated principals (no anonymous access) and over HTTPS. To mitigate security risks, we recommend the following: + Scope the Lambda execution role to the smallest set of permissions possible. + Whenever possible, make sure your Lambda function accesses Amazon S3 through the provided presigned URL. ## Configuring IAM policies S3 access points support AWS Identity and Access Management (IAM) resource policies that allow you to control the use of the access point by resource, user, or other conditions. For more information, see [Configuring IAM policies for Object Lambda Access Points](olap-policies.md). ## Encryption behavior Because Object Lambda Access Points use both Amazon S3 and AWS Lambda, there are differences in encryption behavior. For more information about default S3 encryption behavior, see [Setting default server-side encryption behavior for Amazon S3 buckets](bucket-encryption.md). + When you're using S3 server-side encryption with Object Lambda Access Points, the object is decrypted before being sent to Lambda. After the object is sent to Lambda, it is processed unencrypted (in the case of a `GET` or `HEAD` request). + To prevent the encryption key from being logged, S3 rejects `GET` and `HEAD` requests for objects that are encrypted by using server-side encryption with customer-provided keys (SSE-C). However, the Lambda function might still retrieve these objects if it has access to the client-provided key. + When using S3 client-side encryption with Object Lambda Access Points, make sure that Lambda has access to the encryption key so that it can decrypt and re-encrypt the object. ## Access points security S3 Object Lambda uses two access points, an Object Lambda Access Point and a standard S3 access point, which is referred to as the *supporting access point*. When you make a request to an Object Lambda Access Point, S3 either invokes Lambda on your behalf, or it delegates the request to the supporting access point, depending upon the S3 Object Lambda configuration. When Lambda is invoked for a request S3 generates a presigned URL to your object on your behalf through the supporting access point. Your Lambda function receives this URL as input when the function is invoked. You can set your Lambda function to use this presigned URL to retrieve the original object, instead of invoking S3 directly. By using this model, you can apply better security boundaries to your objects. You can limit direct object access through S3 buckets or S3 access points to a limited set of IAM roles or users. This approach also protects your Lambda functions from being subject to the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html), where a misconfigured function with different permissions than the invoker could allow or deny access to objects when it should not. ## Object Lambda Access Point public access S3 Object Lambda does not allow anonymous or public access because Amazon S3 must authorize your identity to complete any S3 Object Lambda request. When invoking requests through an Object Lambda Access Point, you must have the `lambda:InvokeFunction` permission for the configured Lambda function. Similarly, when invoking other API operations through an Object Lambda Access Point, you must have the required `s3:*` permissions. Without these permissions, requests to invoke Lambda or delegate to S3 will fail with HTTP 403 (Forbidden) errors. All access must be made by authenticated principals. If you require public access, you can use Lambda@Edge as a possible alternative. For more information, see [Customizing at the edge with Lambda@Edge](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html) in the *Amazon CloudFront Developer Guide*. ## Object Lambda Access Point IP addresses The `describe-managed-prefix-lists` subnets support gateway virtual private cloud (VPC) endpoints and are related to the routing table of VPC endpoints. Since Object Lambda Access Point does not support gateway VPC its IP ranges are missing. The missing ranges belong to Amazon S3, but are not supported by gateway VPC endpoints. For more information about `describe-managed-prefix-lists`, see [DescribeManagedPrefixLists](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeManagedPrefixLists.html) in the *Amazon EC2 API Reference* and [AWS IP address ranges](https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html) in the *AWS General Reference*. # Configuring IAM policies for Object Lambda Access Points **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). Amazon S3 access points support AWS Identity and Access Management (IAM) resource policies that you can use to control the use of the access point by resource, user, or other conditions. You can control access through an optional resource policy on your Object Lambda Access Point, or a resource policy on supporting access point. For step-by-step examples, see [Tutorial: Transforming data for your application with S3 Object Lambda](tutorial-s3-object-lambda-uppercase.md) and [Tutorial: Detecting and redacting PII data with S3 Object Lambda and Amazon Comprehend](tutorial-s3-object-lambda-redact-pii.md). The following four resources must have permissions granted to work with Object Lambda Access Points: + The IAM identity, such as user or role. For more information about IAM identities and best practices, see [IAM identities (users, user groups, and roles)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html) in the *IAM User Guide*. + The standard access point connected to an underlying data source such as an S3 bucket or Amazon FSx for OpenZFS volume. When you're working with Object Lambda Access Points, this standard access point is known as a *supporting access point*. + The Object Lambda Access Point. + The AWS Lambda function. **Important** Before you save your policy, make sure to resolve security warnings, errors, general warnings, and suggestions from AWS Identity and Access Management Access Analyzer. IAM Access Analyzer runs policy checks to validate your policy against IAM [policy grammar](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_grammar.html) and [best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). These checks generate findings and provide actionable recommendations to help you author policies that are functional and conform to security best practices. To learn more about validating policies by using IAM Access Analyzer, see [IAM Access Analyzer policy validation](https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-validation.html) in the *IAM User Guide*. To view a list of the warnings, errors, and suggestions that are returned by IAM Access Analyzer, see [IAM Access Analyzer policy check reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-reference-policy-checks.html). The following policy examples assume that you have the following resources: + An Amazon S3 bucket with the following Amazon Resource Name (ARN): `arn:aws:s3:::amzn-s3-demo-bucket1` + An Amazon S3 standard access point on this bucket with the following ARN: `arn:aws:s3:us-east-1:111122223333:accesspoint/my-access-point` + An Object Lambda Access Point with the following ARN: `arn:aws:s3-object-lambda:us-east-1:111122223333:accesspoint/my-object-lambda-ap` + An AWS Lambda function with the following ARN: `arn:aws:lambda:us-east-1:111122223333:function:MyObjectLambdaFunction` **Note** If you're using a Lambda function from your account, you must include the specific function version in your policy statement. In the following example ARN, the version is indicated by *1*: `arn:aws:lambda:us-east-1:111122223333:function:MyObjectLambdaFunction:1` Lambda doesn't support adding IAM policies to the version `$LATEST`. For more information about Lambda function versions, see [Lambda function versions](https://docs.aws.amazon.com/lambda/latest/dg/configuration-versions.html) in the *AWS Lambda Developer Guide*. **Example – Bucket policy that delegates access control to standard access points** The following S3 bucket policy example delegates access control for a bucket to the bucket's standard access points. This policy allows full access to all access points that are owned by the bucket owner's account. Thus, all access to this bucket is controlled by the policies that are attached to its access points. Users can read from the bucket only through an access point, which means that operations can be invoked only through access points. For more information, see [Delegating access control to access points](access-points-policies.md#access-points-delegating-control). **** ``` { "Version":"2012-10-17", "Statement" : [ { "Effect": "Allow", "Principal" : { "AWS":"account-ARN"}, "Action" : "*", "Resource" : [ "arn:aws:s3:::amzn-s3-demo-bucket", "arn:aws:s3:::amzn-s3-demo-bucket/*" ], "Condition": { "StringEquals" : { "s3:DataAccessPointAccount" : "Bucket owner's account ID" } } }] } ``` **Example – IAM policy that grants a user the necessary permissions to use an Object Lambda Access Point** The following IAM policy grants a user permissions to the Lambda function, the standard access point, and the Object Lambda Access Point. **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowLambdaInvocation", "Action": [ "lambda:InvokeFunction" ], "Effect": "Allow", "Resource": "arn:aws:lambda:us-east-1:111122223333:function:MyObjectLambdaFunction:1", "Condition": { "ForAnyValue:StringEquals": { "aws:CalledVia": [ "s3-object-lambda.amazonaws.com" ] } } }, { "Sid": "AllowStandardAccessPointAccess", "Action": [ "s3:Get*", "s3:List*" ], "Effect": "Allow", "Resource": "arn:aws:s3:us-east-1:111122223333:accesspoint/my-access-point/*", "Condition": { "ForAnyValue:StringEquals": { "aws:CalledVia": [ "s3-object-lambda.amazonaws.com" ] } } }, { "Sid": "AllowObjectLambdaAccess", "Action": [ "s3-object-lambda:Get*", "s3-object-lambda:List*" ], "Effect": "Allow", "Resource": "arn:aws:s3-object-lambda:us-east-1:111122223333:accesspoint/my-object-lambda-ap" } ] } ``` ## Enable permissions for Lambda execution roles When `GET` requests are made to an Object Lambda Access Point, your Lambda function needs permission to send data to S3 Object Lambda Access Point. This permission is provided by enabling the `s3-object-lambda:WriteGetObjectResponse` permission on your Lambda function's execution role. You can create a new execution role or update an existing one. **Note** Your function needs the `s3-object-lambda:WriteGetObjectResponse` permission only if you're making a `GET` request. **To create an execution role in the IAM console** 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**. 1. Choose **Create role**. 1. Under **Common use cases**, choose **Lambda**. 1. Choose **Next**. 1. On the **Add permissions** page, search for the AWS managed policy [https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/service-role/AmazonS3ObjectLambdaExecutionRolePolicy$serviceLevelSummary](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/service-role/AmazonS3ObjectLambdaExecutionRolePolicy$serviceLevelSummary), and then select the check box beside the policy name. This policy should contain the `s3-object-lambda:WriteGetObjectResponse` Action. 1. Choose **Next**. 1. On the **Name, review, and create** page, for **Role name**, enter **s3-object-lambda-role**. 1. (Optional) Add a description and tags for this role. 1. Choose **Create role**. 1. Apply the newly created **s3-object-lambda-role** as your Lambda function's execution role. This can be done during or after Lambda function creation in the Lambda console. For more information about execution roles, see [Lambda execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html) in the *AWS Lambda Developer Guide*. ## Using context keys with Object Lambda Access Points S3 Object Lambda will evaluate context keys such as `s3-object-lambda:TlsVersion` or `s3-object-lambda:AuthType` that are related to the connection or signing of the request. All other context keys, such as `s3:prefix`, are evaluated by Amazon S3. ## Object Lambda Access Point CORS support When S3 Object Lambda receives a request from a browser or the request includes an `Origin` header, S3 Object Lambda always adds an `"AllowedOrigins":"*"` header field. For more information, see [Using cross-origin resource sharing (CORS)](cors.md). # Writing Lambda functions for S3 Object Lambda Access Points **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). This section details how to write AWS Lambda functions for use with Amazon S3 Object Lambda Access Points. To learn about complete end-to-end procedures for some S3 Object Lambda tasks, see the following: + [Tutorial: Transforming data for your application with S3 Object Lambda](tutorial-s3-object-lambda-uppercase.md) + [Tutorial: Detecting and redacting PII data with S3 Object Lambda and Amazon Comprehend](tutorial-s3-object-lambda-redact-pii.md) + [Tutorial: Using S3 Object Lambda to dynamically watermark images as they are retrieved](https://aws.amazon.com/getting-started/hands-on/amazon-s3-object-lambda-to-dynamically-watermark-images/?ref=docs_gateway/amazons3/olap-writing-lambda.html) **Topics** + [ ## Working with `GetObject` requests in Lambda ](#olap-getobject-response) + [ ## Working with `HeadObject` requests in Lambda ](#olap-headobject) + [ ## Working with `ListObjects` requests in Lambda ](#olap-listobjects) + [ ## Working with `ListObjectsV2` requests in Lambda ](#olap-listobjectsv2) + [ # Event context format and usage ](olap-event-context.md) + [ # Working with Range and partNumber headers ](range-get-olap.md) ## Working with `GetObject` requests in Lambda This section assumes that your Object Lambda Access Point is configured to call the Lambda function for `GetObject`. S3 Object Lambda includes the Amazon S3 API operation, `WriteGetObjectResponse`, which enables the Lambda function to provide customized data and response headers to the `GetObject` caller. `WriteGetObjectResponse` gives you extensive control over the status code, response headers, and response body, based on your processing needs. You can use `WriteGetObjectResponse` to respond with the whole transformed object, portions of the transformed object, or other responses based on the context of your application. The following section shows unique examples of using the `WriteGetObjectResponse` API operation. + **Example 1:** Respond with HTTP status code 403 (Forbidden) + **Example 2:** Respond with a transformed image + **Example 3:** Stream compressed content **Example 1: Respond with HTTP status code 403 (Forbidden) ** You can use `WriteGetObjectResponse` to respond with the HTTP status code 403 (Forbidden) based on the content of the object. ------ #### [ Java ] ``` package com.amazon.s3.objectlambda; import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda.runtime.events.S3ObjectLambdaEvent; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.WriteGetObjectResponseRequest; import java.io.ByteArrayInputStream; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; public class Example1 { public void handleRequest(S3ObjectLambdaEvent event, Context context) throws Exception { S3Client s3Client = S3Client.builder().build(); // Check to see if the request contains all of the necessary information. // If it does not, send a 4XX response and a custom error code and message. // Otherwise, retrieve the object from S3 and stream it // to the client unchanged. var tokenIsNotPresent = !event.getUserRequest().getHeaders().containsKey("requiredToken"); if (tokenIsNotPresent) { s3Client.writeGetObjectResponse(WriteGetObjectResponseRequest.builder() .requestRoute(event.outputRoute()) .requestToken(event.outputToken()) .statusCode(403) .contentLength(0L) .errorCode("MissingRequiredToken") .errorMessage("The required token was not present in the request.") .build(), RequestBody.fromInputStream(new ByteArrayInputStream(new byte[0]), 0L)); return; } // Prepare the presigned URL for use and make the request to S3. HttpClient httpClient = HttpClient.newBuilder().build(); var presignedResponse = httpClient.send( HttpRequest.newBuilder(new URI(event.inputS3Url())).GET().build(), HttpResponse.BodyHandlers.ofInputStream()); // Stream the original bytes back to the caller. s3Client.writeGetObjectResponse(WriteGetObjectResponseRequest.builder() .requestRoute(event.outputRoute()) .requestToken(event.outputToken()) .build(), RequestBody.fromInputStream(presignedResponse.body(), presignedResponse.headers().firstValueAsLong("content-length").orElse(-1L))); } } ``` ------ #### [ Python ] ``` import boto3 import requests def handler(event, context): s3 = boto3.client('s3') """ Retrieve the operation context object from the event. This object indicates where the WriteGetObjectResponse request should be delivered and contains a presigned URL in 'inputS3Url' where we can download the requested object from. The 'userRequest' object has information related to the user who made this 'GetObject' request to S3 Object Lambda. """ get_context = event["getObjectContext"] user_request_headers = event["userRequest"]["headers"] route = get_context["outputRoute"] token = get_context["outputToken"] s3_url = get_context["inputS3Url"] # Check for the presence of a 'CustomHeader' header and deny or allow based on that header. is_token_present = "SuperSecretToken" in user_request_headers if is_token_present: # If the user presented our custom 'SuperSecretToken' header, we send the requested object back to the user. response = requests.get(s3_url) s3.write_get_object_response(RequestRoute=route, RequestToken=token, Body=response.content) else: # If the token is not present, we send an error back to the user. s3.write_get_object_response(RequestRoute=route, RequestToken=token, StatusCode=403, ErrorCode="NoSuperSecretTokenFound", ErrorMessage="The request was not secret enough.") # Gracefully exit the Lambda function. return { 'status_code': 200 } ``` ------ #### [ Node.js ] ``` const { S3 } = require('aws-sdk'); const axios = require('axios').default; exports.handler = async (event) => { const s3 = new S3(); // Retrieve the operation context object from the event. This object indicates where the WriteGetObjectResponse request // should be delivered and contains a presigned URL in 'inputS3Url' where we can download the requested object from. // The 'userRequest' object has information related to the user who made this 'GetObject' request to S3 Object Lambda. const { userRequest, getObjectContext } = event; const { outputRoute, outputToken, inputS3Url } = getObjectContext; // Check for the presence of a 'CustomHeader' header and deny or allow based on that header. const isTokenPresent = Object .keys(userRequest.headers) .includes("SuperSecretToken"); if (!isTokenPresent) { // If the token is not present, we send an error back to the user. The 'await' in front of the request // indicates that we want to wait for this request to finish sending before moving on. await s3.writeGetObjectResponse({ RequestRoute: outputRoute, RequestToken: outputToken, StatusCode: 403, ErrorCode: "NoSuperSecretTokenFound", ErrorMessage: "The request was not secret enough.", }).promise(); } else { // If the user presented our custom 'SuperSecretToken' header, we send the requested object back to the user. // Again, note the presence of 'await'. const presignedResponse = await axios.get(inputS3Url); await s3.writeGetObjectResponse({ RequestRoute: outputRoute, RequestToken: outputToken, Body: presignedResponse.data, }).promise(); } // Gracefully exit the Lambda function. return { statusCode: 200 }; } ``` ------ **Example 2: Respond with a transformed image** When performing an image transformation, you might find that you need all the bytes of the source object before you can start processing them. In this case, your `WriteGetObjectResponse` request returns the whole object to the requesting application in one call. ------ #### [ Java ] ``` package com.amazon.s3.objectlambda; import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda.runtime.events.S3ObjectLambdaEvent; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.WriteGetObjectResponseRequest; import javax.imageio.ImageIO; import java.awt.image.BufferedImage; import java.awt.Image; import java.io.ByteArrayOutputStream; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; public class Example2V2 { private static final int HEIGHT = 250; private static final int WIDTH = 250; public void handleRequest(S3ObjectLambdaEvent event, Context context) throws Exception { S3Client s3Client = S3Client.builder().build(); HttpClient httpClient = HttpClient.newBuilder().build(); // Prepare the presigned URL for use and make the request to S3. var presignedResponse = httpClient.send( HttpRequest.newBuilder(new URI(event.inputS3Url())).GET().build(), HttpResponse.BodyHandlers.ofInputStream()); // The entire image is loaded into memory here so that we can resize it. // Once the resizing is completed, we write the bytes into the body // of the WriteGetObjectResponse request. var originalImage = ImageIO.read(presignedResponse.body()); var resizingImage = originalImage.getScaledInstance(WIDTH, HEIGHT, Image.SCALE_DEFAULT); var resizedImage = new BufferedImage(WIDTH, HEIGHT, BufferedImage.TYPE_INT_RGB); resizedImage.createGraphics().drawImage(resizingImage, 0, 0, WIDTH, HEIGHT, null); var baos = new ByteArrayOutputStream(); ImageIO.write(resizedImage, "png", baos); // Stream the bytes back to the caller. s3Client.writeGetObjectResponse(WriteGetObjectResponseRequest.builder() .requestRoute(event.outputRoute()) .requestToken(event.outputToken()) .build(), RequestBody.fromBytes(baos.toByteArray())); } } ``` ------ #### [ Python ] ``` import boto3 import requests import io from PIL import Image def handler(event, context): """ Retrieve the operation context object from the event. This object indicates where the WriteGetObjectResponse request should be delivered and has a presigned URL in 'inputS3Url' where we can download the requested object from. The 'userRequest' object has information related to the user who made this 'GetObject' request to S3 Object Lambda. """ get_context = event["getObjectContext"] route = get_context["outputRoute"] token = get_context["outputToken"] s3_url = get_context["inputS3Url"] """ In this case, we're resizing .png images that are stored in S3 and are accessible through the presigned URL 'inputS3Url'. """ image_request = requests.get(s3_url) image = Image.open(io.BytesIO(image_request.content)) image.thumbnail((256,256), Image.ANTIALIAS) transformed = io.BytesIO() image.save(transformed, "png") # Send the resized image back to the client. s3 = boto3.client('s3') s3.write_get_object_response(Body=transformed.getvalue(), RequestRoute=route, RequestToken=token) # Gracefully exit the Lambda function. return { 'status_code': 200 } ``` ------ #### [ Node.js ] ``` const { S3 } = require('aws-sdk'); const axios = require('axios').default; const sharp = require('sharp'); exports.handler = async (event) => { const s3 = new S3(); // Retrieve the operation context object from the event. This object indicates where the WriteGetObjectResponse request // should be delivered and has a presigned URL in 'inputS3Url' where we can download the requested object from. const { getObjectContext } = event; const { outputRoute, outputToken, inputS3Url } = getObjectContext; // In this case, we're resizing .png images that are stored in S3 and are accessible through the presigned URL // 'inputS3Url'. const { data } = await axios.get(inputS3Url, { responseType: 'arraybuffer' }); // Resize the image. const resized = await sharp(data) .resize({ width: 256, height: 256 }) .toBuffer(); // Send the resized image back to the client. await s3.writeGetObjectResponse({ RequestRoute: outputRoute, RequestToken: outputToken, Body: resized, }).promise(); // Gracefully exit the Lambda function. return { statusCode: 200 }; } ``` ------ **Example 3: Stream compressed content** When you're compressing objects, compressed data is produced incrementally. Consequently, you can use your `WriteGetObjectResponse` request to return the compressed data as soon as it's ready. As shown in this example, you don't need to know the length of the completed transformation. ------ #### [ Java ] ``` package com.amazon.s3.objectlambda; import com.amazonaws.services.lambda.runtime.events.S3ObjectLambdaEvent; import com.amazonaws.services.lambda.runtime.Context; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.WriteGetObjectResponseRequest; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; public class Example3 { public void handleRequest(S3ObjectLambdaEvent event, Context context) throws Exception { S3Client s3Client = S3Client.builder().build(); HttpClient httpClient = HttpClient.newBuilder().build(); // Request the original object from S3. var presignedResponse = httpClient.send( HttpRequest.newBuilder(new URI(event.inputS3Url())).GET().build(), HttpResponse.BodyHandlers.ofInputStream()); // Consume the incoming response body from the presigned request, // apply our transformation on that data, and emit the transformed bytes // into the body of the WriteGetObjectResponse request as soon as they're ready. // This example compresses the data from S3, but any processing pertinent // to your application can be performed here. var bodyStream = new GZIPCompressingInputStream(presignedResponse.body()); // Stream the bytes back to the caller. s3Client.writeGetObjectResponse(WriteGetObjectResponseRequest.builder() .requestRoute(event.outputRoute()) .requestToken(event.outputToken()) .build(), RequestBody.fromInputStream(bodyStream, presignedResponse.headers().firstValueAsLong("content-length").orElse(-1L))); } } ``` ------ #### [ Python ] ``` import boto3 import requests import zlib from botocore.config import Config """ A helper class to work with content iterators. Takes an interator and compresses the bytes that come from it. It implements 'read' and '__iter__' so that the SDK can stream the response. """ class Compress: def __init__(self, content_iter): self.content = content_iter self.compressed_obj = zlib.compressobj() def read(self, _size): for data in self.__iter__() return data def __iter__(self): while True: data = next(self.content) chunk = self.compressed_obj.compress(data) if not chunk: break yield chunk yield self.compressed_obj.flush() def handler(event, context): """ Setting the 'payload_signing_enabled' property to False allows us to send a streamed response back to the client. in this scenario, a streamed response means that the bytes are not buffered into memory as we're compressing them, but instead are sent straight to the user. """ my_config = Config( region_name='eu-west-1', signature_version='s3v4', s3={ "payload_signing_enabled": False } ) s3 = boto3.client('s3', config=my_config) """ Retrieve the operation context object from the event. This object indicates where the WriteGetObjectResponse request should be delivered and has a presigned URL in 'inputS3Url' where we can download the requested object from. The 'userRequest' object has information related to the user who made this 'GetObject' request to S3 Object Lambda. """ get_context = event["getObjectContext"] route = get_context["outputRoute"] token = get_context["outputToken"] s3_url = get_context["inputS3Url"] # Compress the 'get' request stream. with requests.get(s3_url, stream=True) as r: compressed = Compress(r.iter_content()) # Send the stream back to the client. s3.write_get_object_response(Body=compressed, RequestRoute=route, RequestToken=token, ContentType="text/plain", ContentEncoding="gzip") # Gracefully exit the Lambda function. return {'status_code': 200} ``` ------ #### [ Node.js ] ``` const { S3 } = require('aws-sdk'); const axios = require('axios').default; const zlib = require('zlib'); exports.handler = async (event) => { const s3 = new S3(); // Retrieve the operation context object from the event. This object indicates where the WriteGetObjectResponse request // should be delivered and has a presigned URL in 'inputS3Url' where we can download the requested object from. const { getObjectContext } = event; const { outputRoute, outputToken, inputS3Url } = getObjectContext; // Download the object from S3 and process it as a stream, because it might be a huge object and we don't want to // buffer it in memory. Note the use of 'await' because we want to wait for 'writeGetObjectResponse' to finish // before we can exit the Lambda function. await axios({ method: 'GET', url: inputS3Url, responseType: 'stream', }).then( // Gzip the stream. response => response.data.pipe(zlib.createGzip()) ).then( // Finally send the gzip-ed stream back to the client. stream => s3.writeGetObjectResponse({ RequestRoute: outputRoute, RequestToken: outputToken, Body: stream, ContentType: "text/plain", ContentEncoding: "gzip", }).promise() ); // Gracefully exit the Lambda function. return { statusCode: 200 }; } ``` ------ **Note** Although S3 Object Lambda allows up to 60 seconds to send a complete response to the caller through the `WriteGetObjectResponse` request, the actual amount of time available might be less. For example, your Lambda function timeout might be less than 60 seconds. In other cases, the caller might have more stringent timeouts. For the original caller to receive a response other than HTTP status code 500 (Internal Server Error), the `WriteGetObjectResponse` call must be completed. If the Lambda function returns, with an exception or otherwise, before the `WriteGetObjectResponse` API operation is called, the original caller receives a 500 (Internal Server Error) response. Exceptions thrown during the time it takes to complete the response result in truncated responses to the caller. If the Lambda function receives an HTTP status code 200 (OK) response from the `WriteGetObjectResponse` API call, then the original caller has sent the complete request. The Lambda function's response, whether an exception is thrown or not, is ignored by S3 Object Lambda. When calling the `WriteGetObjectResponse` API operation, Amazon S3 requires the route and request token from the event context. For more information, see [Event context format and usage](olap-event-context.md). The route and request token parameters are required to connect the `WriteGetObjectResult` response with the original caller. Even though it is always appropriate to retry 500 (Internal Server Error) responses, because the request token is a single-use token, subsequent attempts to use it might result in HTTP status code 400 (Bad Request) responses. Although the call to `WriteGetObjectResponse` with the route and request tokens doesn't need to be made from the invoked Lambda function, it must be made by an identity in the same account. The call also must be completed before the Lambda function finishes execution. ## Working with `HeadObject` requests in Lambda This section assumes that your Object Lambda Access Point is configured to call the Lambda function for `HeadObject`. Lambda will receive a JSON payload that contains a key called `headObjectContext`. Inside the context, there is a single property called `inputS3Url`, which is a presigned URL for the supporting access point for `HeadObject`. The presigned URL will include the following properties if they're specified: + `versionId` (in the query parameters) + `requestPayer` (in the `x-amz-request-payer` header) + `expectedBucketOwner` (in the `x-amz-expected-bucket-owner` header) Other properties won't be presigned, and thus won't be included. Non-signed options sent as headers can be added manually to the request when calling the presigned URL that's found in the `userRequest` headers. Server-side encryption options are not supported for `HeadObject`. For the request syntax URI parameters, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) in the *Amazon Simple Storage Service API Reference*. The following example shows a Lambda JSON input payload for `HeadObject`. ``` { "xAmzRequestId": "requestId", "**headObjectContext**": { "**inputS3Url**": "https://my-s3-ap-111122223333.s3-accesspoint.us-east-1.amazonaws.com/example?X-Amz-Security-Token=" }, "configuration": { "accessPointArn": "arn:aws:s3-object-lambda:us-east-1:111122223333:accesspoint/example-object-lambda-ap", "supportingAccessPointArn": "arn:aws:s3:us-east-1:111122223333:accesspoint/example-ap", "payload": "{}" }, "userRequest": { "url": "https://object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com/example", "headers": { "Host": "object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com", "Accept-Encoding": "identity", "X-Amz-Content-SHA256": "e3b0c44298fc1example" } }, "userIdentity": { "type": "AssumedRole", "principalId": "principalId", "arn": "arn:aws:sts::111122223333:assumed-role/Admin/example", "accountId": "111122223333", "accessKeyId": "accessKeyId", "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "Wed Mar 10 23:41:52 UTC 2021" }, "sessionIssuer": { "type": "Role", "principalId": "principalId", "arn": "arn:aws:iam::111122223333:role/Admin", "accountId": "111122223333", "userName": "Admin" } } }, "protocolVersion": "1.00" } ``` Your Lambda function should return a JSON object that contains the headers and values that will be returned for the `HeadObject` call. The following example shows the structure of the Lambda response JSON for `HeadObject`. ``` { "statusCode": ; // Required "errorCode": ; "errorMessage": ; "headers": { "Accept-Ranges": , "x-amz-archive-status": , "x-amz-server-side-encryption-bucket-key-enabled": , "Cache-Control": , "Content-Disposition": , "Content-Encoding": , "Content-Language": , "Content-Length": , // Required "Content-Type": , "x-amz-delete-marker": , "ETag": , "Expires": , "x-amz-expiration": , "Last-Modified": , "x-amz-missing-meta": , "x-amz-object-lock-mode": , "x-amz-object-lock-legal-hold": , "x-amz-object-lock-retain-until-date": , "x-amz-mp-parts-count": , "x-amz-replication-status": , "x-amz-request-charged": , "x-amz-restore": , "x-amz-server-side-encryption": , "x-amz-server-side-encryption-customer-algorithm": , "x-amz-server-side-encryption-aws-kms-key-id": , "x-amz-server-side-encryption-customer-key-MD5": , "x-amz-storage-class": , "x-amz-tagging-count": , "x-amz-version-id": , : , // user-defined metadata "x-amz-meta-meta1": , // example of the user-defined metadata header, it will need the x-amz-meta prefix "x-amz-meta-meta2": ... }; } ``` The following example shows how to use the presigned URL to populate your response by modifying the header values as needed before returning the JSON. ------ #### [ Python ] ``` import requests def lambda_handler(event, context): print(event) # Extract the presigned URL from the input. s3_url = event["headObjectContext"]["inputS3Url"] # Get the head of the object from S3. response = requests.head(s3_url) # Return the error to S3 Object Lambda (if applicable). if (response.status_code >= 400): return { "statusCode": response.status_code, "errorCode": "RequestFailure", "errorMessage": "Request to S3 failed" } # Store the headers in a dictionary. response_headers = dict(response.headers) # This obscures Content-Type in a transformation, it is optional to add response_headers["Content-Type"] = "" # Return the headers to S3 Object Lambda. return { "statusCode": response.status_code, "headers": response_headers } ``` ------ ## Working with `ListObjects` requests in Lambda This section assumes that your Object Lambda Access Point is configured to call the Lambda function for `ListObjects`. Lambda will receive the JSON payload with a new object named `listObjectsContext`. `listObjectsContext`contains a single property, `inputS3Url`, which is a presigned URL for the supporting access point for `ListObjects`. Unlike `GetObject` and `HeadObject`, the presigned URL will include the following properties if they're specified: + All the query parameters + `requestPayer` (in the `x-amz-request-payer` header) + `expectedBucketOwner` (in the `x-amz-expected-bucket-owner` header) For the request syntax URI parameters, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html) in the *Amazon Simple Storage Service API Reference*. **Important** We recommend that you use the newer version, [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html), when developing applications. For backward compatibility, Amazon S3 continues to support `ListObjects`. The following example shows the Lambda JSON input payload for `ListObjects`. ``` { "xAmzRequestId": "requestId", "**listObjectsContext**": { "**inputS3Url**": "https://my-s3-ap-111122223333.s3-accesspoint.us-east-1.amazonaws.com/?X-Amz-Security-Token=", }, "configuration": { "accessPointArn": "arn:aws:s3-object-lambda:us-east-1:111122223333:accesspoint/example-object-lambda-ap", "supportingAccessPointArn": "arn:aws:s3:us-east-1:111122223333:accesspoint/example-ap", "payload": "{}" }, "userRequest": { "url": "https://object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com/example", "headers": { "Host": "object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com", "Accept-Encoding": "identity", "X-Amz-Content-SHA256": "e3b0c44298fc1example" } }, "userIdentity": { "type": "AssumedRole", "principalId": "principalId", "arn": "arn:aws:sts::111122223333:assumed-role/Admin/example", "accountId": "111122223333", "accessKeyId": "accessKeyId", "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "Wed Mar 10 23:41:52 UTC 2021" }, "sessionIssuer": { "type": "Role", "principalId": "principalId", "arn": "arn:aws:iam::111122223333:role/Admin", "accountId": "111122223333", "userName": "Admin" } } }, "protocolVersion": "1.00" } ``` Your Lambda function should return a JSON object that contains the status code, list XML result, or error information that will be returned from S3 Object Lambda. S3 Object Lambda does not process or validate `listResultXml`, but instead forwards it to `ListObjects` caller. For `listBucketResult`, S3 Object Lambda expects certain properties to be of a specific type and will throw exceptions if it cannot parse them. `listResultXml` and `listBucketResult` can not be provided at the same time. The following example demonstrates how to use the presigned URL to call Amazon S3 and use the result to populate a response, including error checking. ------ #### [ Python ] ``` import requests import xmltodict def lambda_handler(event, context): # Extract the presigned URL from the input. s3_url = event["listObjectsContext"]["inputS3Url"] # Get the head of the object from Amazon S3. response = requests.get(s3_url) # Return the error to S3 Object Lambda (if applicable). if (response.status_code >= 400): error = xmltodict.parse(response.content) return { "statusCode": response.status_code, "errorCode": error["Error"]["Code"], "errorMessage": error["Error"]["Message"] } # Store the XML result in a dict. response_dict = xmltodict.parse(response.content) # This obscures StorageClass in a transformation, it is optional to add for item in response_dict['ListBucketResult']['Contents']: item['StorageClass'] = "" # Convert back to XML. listResultXml = xmltodict.unparse(response_dict) # Create response with listResultXml. response_with_list_result_xml = { 'statusCode': 200, 'listResultXml': listResultXml } # Create response with listBucketResult. response_dict['ListBucketResult'] = sanitize_response_dict(response_dict['ListBucketResult']) response_with_list_bucket_result = { 'statusCode': 200, 'listBucketResult': response_dict['ListBucketResult'] } # Return the list to S3 Object Lambda. # Can return response_with_list_result_xml or response_with_list_bucket_result return response_with_list_result_xml # Converting the response_dict's key to correct casing def sanitize_response_dict(response_dict: dict): new_response_dict = dict() for key, value in response_dict.items(): new_key = key[0].lower() + key[1:] if key != "ID" else 'id' if type(value) == list: newlist = [] for element in value: if type(element) == type(dict()): element = sanitize_response_dict(element) newlist.append(element) value = newlist elif type(value) == dict: value = sanitize_response_dict(value) new_response_dict[new_key] = value return new_response_dict ``` ------ The following example shows the structure of the Lambda response JSON for `ListObjects`. ``` { "statusCode": ; // Required "errorCode": ; "errorMessage": ; "listResultXml": ; // This can also be Error XML string in case S3 returned error response when calling the pre-signed URL "listBucketResult": { // listBucketResult can be provided instead of listResultXml, however they can not both be provided in the JSON response "name": , // Required for 'listBucketResult' "prefix": , "marker": , "nextMarker": , "maxKeys": , // Required for 'listBucketResult' "delimiter": , "encodingType": "isTruncated": , // Required for 'listBucketResult' "contents": [ { "key": , // Required for 'content' "lastModified": , "eTag": , "checksumAlgorithm": , // CRC32, CRC32C, SHA1, SHA256 "size": , // Required for 'content' "owner": { "displayName": , // Required for 'owner' "id": , // Required for 'owner' }, "storageClass": }, ... ], "commonPrefixes": [ { "prefix": // Required for 'commonPrefix' }, ... ], } } ``` ## Working with `ListObjectsV2` requests in Lambda This section assumes that your Object Lambda Access Point is configured to call the Lambda function for `ListObjectsV2`. Lambda will receive the JSON payload with a new object named `listObjectsV2Context`. `listObjectsV2Context` contains a single property, `inputS3Url`, which is a presigned URL for the supporting access point for `ListObjectsV2`. Unlike `GetObject` and `HeadObject`, the presigned URL will include the following properties, if they're specified: + All the query parameters + `requestPayer` (in the `x-amz-request-payer` header) + `expectedBucketOwner` (in the `x-amz-expected-bucket-owner` header) For the request syntax URI parameters, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) in the *Amazon Simple Storage Service API Reference*. The following example shows the Lambda JSON input payload for `ListObjectsV2`. ``` { "xAmzRequestId": "requestId", "**listObjectsV2Context**": { "**inputS3Url**": "https://my-s3-ap-111122223333.s3-accesspoint.us-east-1.amazonaws.com/?list-type=2&X-Amz-Security-Token=", }, "configuration": { "accessPointArn": "arn:aws:s3-object-lambda:us-east-1:111122223333:accesspoint/example-object-lambda-ap", "supportingAccessPointArn": "arn:aws:s3:us-east-1:111122223333:accesspoint/example-ap", "payload": "{}" }, "userRequest": { "url": "https://object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com/example", "headers": { "Host": "object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com", "Accept-Encoding": "identity", "X-Amz-Content-SHA256": "e3b0c44298fc1example" } }, "userIdentity": { "type": "AssumedRole", "principalId": "principalId", "arn": "arn:aws:sts::111122223333:assumed-role/Admin/example", "accountId": "111122223333", "accessKeyId": "accessKeyId", "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "Wed Mar 10 23:41:52 UTC 2021" }, "sessionIssuer": { "type": "Role", "principalId": "principalId", "arn": "arn:aws:iam::111122223333:role/Admin", "accountId": "111122223333", "userName": "Admin" } } }, "protocolVersion": "1.00" } ``` Your Lambda function should return a JSON object that contains the status code, list XML result, or error information that will be returned from S3 Object Lambda. S3 Object Lambda does not process or validate `listResultXml`, but instead forwards it to `ListObjectsV2` caller. For `listBucketResult`, S3 Object Lambda expects certain properties to be of a specific type and will throw exceptions if it cannot parse them. `listResultXml` and `listBucketResult` can not be provided at the same time. The following example demonstrates how to use the presigned URL to call Amazon S3 and use the result to populate a response, including error checking. ------ #### [ Python ] ``` import requests import xmltodict def lambda_handler(event, context): # Extract the presigned URL from the input. s3_url = event["listObjectsV2Context"]["inputS3Url"] # Get the head of the object from Amazon S3. response = requests.get(s3_url) # Return the error to S3 Object Lambda (if applicable). if (response.status_code >= 400): error = xmltodict.parse(response.content) return { "statusCode": response.status_code, "errorCode": error["Error"]["Code"], "errorMessage": error["Error"]["Message"] } # Store the XML result in a dict. response_dict = xmltodict.parse(response.content) # This obscures StorageClass in a transformation, it is optional to add for item in response_dict['ListBucketResult']['Contents']: item['StorageClass'] = "" # Convert back to XML. listResultXml = xmltodict.unparse(response_dict) # Create response with listResultXml. response_with_list_result_xml = { 'statusCode': 200, 'listResultXml': listResultXml } # Create response with listBucketResult. response_dict['ListBucketResult'] = sanitize_response_dict(response_dict['ListBucketResult']) response_with_list_bucket_result = { 'statusCode': 200, 'listBucketResult': response_dict['ListBucketResult'] } # Return the list to S3 Object Lambda. # Can return response_with_list_result_xml or response_with_list_bucket_result return response_with_list_result_xml # Converting the response_dict's key to correct casing def sanitize_response_dict(response_dict: dict): new_response_dict = dict() for key, value in response_dict.items(): new_key = key[0].lower() + key[1:] if key != "ID" else 'id' if type(value) == list: newlist = [] for element in value: if type(element) == type(dict()): element = sanitize_response_dict(element) newlist.append(element) value = newlist elif type(value) == dict: value = sanitize_response_dict(value) new_response_dict[new_key] = value return new_response_dict ``` ------ The following example shows the structure of the Lambda response JSON for `ListObjectsV2`. ``` { "statusCode": ; // Required "errorCode": ; "errorMessage": ; "listResultXml": ; // This can also be Error XML string in case S3 returned error response when calling the pre-signed URL "listBucketResult": { // listBucketResult can be provided instead of listResultXml, however they can not both be provided in the JSON response "name": , // Required for 'listBucketResult' "prefix": , "startAfter": , "continuationToken": , "nextContinuationToken": , "keyCount": , // Required for 'listBucketResult' "maxKeys": , // Required for 'listBucketResult' "delimiter": , "encodingType": "isTruncated": , // Required for 'listBucketResult' "contents": [ { "key": , // Required for 'content' "lastModified": , "eTag": , "checksumAlgorithm": , // CRC32, CRC32C, SHA1, SHA256 "size": , // Required for 'content' "owner": { "displayName": , // Required for 'owner' "id": , // Required for 'owner' }, "storageClass": }, ... ], "commonPrefixes": [ { "prefix": // Required for 'commonPrefix' }, ... ], } } ``` # Event context format and usage **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). Amazon S3 Object Lambda provides context about the request that's being made in the event that's passed to your AWS Lambda function. The following shows an example request. Descriptions of the fields are included after the example. ``` { "xAmzRequestId": "requestId", "getObjectContext": { "inputS3Url": "https://my-s3-ap-111122223333.s3-accesspoint.us-east-1.amazonaws.com/example?X-Amz-Security-Token=", "outputRoute": "io-use1-001", "outputToken": "OutputToken" }, "configuration": { "accessPointArn": "arn:aws:s3-object-lambda:us-east-1:111122223333:accesspoint/example-object-lambda-ap", "supportingAccessPointArn": "arn:aws:s3:us-east-1:111122223333:accesspoint/example-ap", "payload": "{}" }, "userRequest": { "url": "https://object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com/example", "headers": { "Host": "object-lambda-111122223333.s3-object-lambda.us-east-1.amazonaws.com", "Accept-Encoding": "identity", "X-Amz-Content-SHA256": "e3b0c44298fc1example" } }, "userIdentity": { "type": "AssumedRole", "principalId": "principalId", "arn": "arn:aws:sts::111122223333:assumed-role/Admin/example", "accountId": "111122223333", "accessKeyId": "accessKeyId", "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "Wed Mar 10 23:41:52 UTC 2021" }, "sessionIssuer": { "type": "Role", "principalId": "principalId", "arn": "arn:aws:iam::111122223333:role/Admin", "accountId": "111122223333", "userName": "Admin" } } }, "protocolVersion": "1.00" } ``` The following fields are included in the request: + `xAmzRequestId` – The Amazon S3 request ID for this request. We recommend that you log this value to help with debugging. + `getObjectContext` – The input and output details for connections to Amazon S3 and S3 Object Lambda. + `inputS3Url` – A presigned URL that can be used to fetch the original object from Amazon S3. The URL is signed by using the original caller's identity, and that user's permissions will apply when the URL is used. If there are signed headers in the URL, the Lambda function must include these headers in the call to Amazon S3, except for the `Host` header. + `outputRoute` – A routing token that is added to the S3 Object Lambda URL when the Lambda function calls `WriteGetObjectResponse`. + `outputToken` – An opaque token that's used by S3 Object Lambda to match the `WriteGetObjectResponse` call with the original caller. + `configuration` – Configuration information about the Object Lambda Access Point. + `accessPointArn` – The Amazon Resource Name (ARN) of the Object Lambda Access Point that received this request. + `supportingAccessPointArn` – The ARN of the supporting access point that is specified in the Object Lambda Access Point configuration. + `payload` – Custom data that is applied to the Object Lambda Access Point configuration. S3 Object Lambda treats this data as an opaque string, so it might need to be decoded before use. + `userRequest` – Information about the original call to S3 Object Lambda. + `url` – The decoded URL of the request as received by S3 Object Lambda, excluding any authorization-related query parameters. + `headers` – A map of string to strings containing the HTTP headers and their values from the original call, excluding any authorization-related headers. If the same header appears multiple times, the values from each instance of the same header are combined into a comma-delimited list. The case of the original headers is retained in this map. + `userIdentity` – Details about the identity that made the call to S3 Object Lambda. For more information, see [Logging data events for trails](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html) in the *AWS CloudTrail User Guide*. + `type` – The type of identity. + `accountId` – The AWS account to which the identity belongs. + `userName` – The friendly name of the identity that made the call. + `principalId` – The unique identifier for the identity that made the call. + `arn` – The ARN of the principal who made the call. The last section of the ARN contains the user or role that made the call. + `sessionContext` – If the request was made with temporary security credentials, this element provides information about the session that was created for those credentials. + `invokedBy` – The name of the AWS service that made the request, such as Amazon EC2 Auto Scaling or AWS Elastic Beanstalk. + `sessionIssuer` – If the request was made with temporary security credentials, this element provides information about how the credentials were obtained. + `protocolVersion` – The version ID of the context provided. The format of this field is `{Major Version}.{Minor Version}`. The minor version numbers are always two-digit numbers. Any removal or change to the semantics of a field necessitates a major version bump and requires active opt-in. Amazon S3 can add new fields at any time, at which point you might experience a minor version bump. Because of the nature of software rollouts, you might see multiple minor versions in use at once. # Working with Range and partNumber headers **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). When working with large objects in Amazon S3 Object Lambda, you can use the `Range` HTTP header to download a specified byte range from an object. To fetch different byte ranges from within the same object, you can use concurrent connections to Amazon S3. You can also specify the `partNumber` parameter (an integer between 1 and 10,000), which performs a ranged request for the specified part of the object. Because there are multiple ways that you might want to handle a request that includes the `Range` or `partNumber` parameters, S3 Object Lambda doesn't apply these parameters to the transformed object. Instead, your AWS Lambda function must implement this functionality as needed for your application. To use the `Range` and `partNumber` parameters with S3 Object Lambda, you do the following: + Enable these parameters in your Object Lambda Access Point configuration. + Write a Lambda function that can handle requests that include these parameters. The following steps describe how to accomplish this. ## Step 1: Configure your Object Lambda Access Point By default, Object Lambda Access Points respond with an HTTP status code 501 (Not Implemented) error to any `GetObject` or `HeadObject` request that contains a `Range` or `partNumber` parameter, either in the headers or query parameters. To enable an Object Lambda Access Point to accept such requests, you must include `GetObject-Range`, `GetObject-PartNumber`, `HeadObject-Range`, or `HeadObject-PartNumber` in the `AllowedFeatures` section of your Object Lambda Access Point configuration. For more information about updating your Object Lambda Access Point configuration, see [Creating Object Lambda Access Points](olap-create.md). ## Step 2: Implement `Range` or `partNumber` handling in your Lambda function When your Object Lambda Access Point invokes your Lambda function with a ranged `GetObject` or `HeadObject` request, the `Range` or `partNumber` parameter is included in the event context. The location of the parameter in the event context depends on which parameter was used and how it was included in the original request to the Object Lambda Access Point, as explained in the following table. | Parameter | Event context location | | --- | --- | | `Range` (header) | `userRequest.headers.Range` | | `Range` (query parameter) | `userRequest.url` (query parameter `Range`) | | `partNumber` | `userRequest.url` (query parameter `partNumber`) | **Important** The provided presigned URL for your Object Lambda Access Point doesn't contain the `Range` or `partNumber` parameter from the original request. See the following options on how to handle these parameters in your AWS Lambda function. After you extract the `Range` or `partNumber` value, you can take one of the following approaches, based on your application's needs: 1. **Map the requested `Range` or `partNumber` to the transformed object (recommended).** The most reliable way to handle `Range` or `partNumber` requests is to do the following: + Retrieve the full object from Amazon S3. + Transform the object. + Apply the requested `Range` or `partNumber` parameters to the transformed object. To do this, use the provided presigned URL to fetch the entire object from Amazon S3 and then process the object as needed. For an example Lambda function that processes a `Range` parameter in this way, see [ this sample](https://github.com/aws-samples/amazon-s3-object-lambda-default-configuration/blob/main/function/nodejs_20_x/src/response/range_mapper.ts) in the AWS Samples GitHub repository. 1. **Map the requested `Range` to the presigned URL.** In some cases, your Lambda function can map the requested `Range` directly to the presigned URL to retrieve only part of the object from Amazon S3. This approach is appropriate only if your transformation meets both of the following criteria: 1. Your transformation function can be applied to partial object ranges. 1. Applying the `Range` parameter before or after the transformation function results in the same transformed object. For example, a transformation function that converts all characters in an ASCII-encoded object to uppercase meets both of the preceding criteria. The transformation can be applied to part of an object, and applying the `Range` parameter before the transformation achieves the same result as applying it after the transformation. By contrast, a function that reverses the characters in an ASCII-encoded object doesn't meet these criteria. Such a function meets criterion 1, because it can be applied to partial object ranges. However, it doesn't meet criterion 2, because applying the `Range` parameter before the transformation achieves different results than applying the parameter after the transformation. Consider a request to apply the function to the first three characters of an object with the contents `abcdefg`. Applying the `Range` parameter before the transformation retrieves only `abc` and then reverses the data, returning `cba`. But if the parameter is applied after the transformation, the function retrieves the entire object, reverses it, and then applies the `Range` parameter, returning `gfe`. Because these results are different, this function should not apply the `Range` parameter when retrieving the object from Amazon S3. Instead, it should retrieve the entire object, perform the transformation, and only then apply the `Range` parameter. **Warning** In many cases, applying the `Range` parameter to the presigned URL will result in unexpected behavior by the Lambda function or the requesting client. Unless you are sure that your application will work properly when retrieving only a partial object from Amazon S3, we recommend that you retrieve and transform full objects as described earlier in approach A. If your application meets the criteria described earlier in approach B, you can simplify your AWS Lambda function by fetching only the requested object range and then running your transformation on that range. The following Java code example demonstrates how to do the following: + Retrieve the `Range` header from the `GetObject` request. + Add the `Range` header to the presigned URL that Lambda can use to retrieve the requested range from Amazon S3. ``` private HttpRequest.Builder applyRangeHeader(ObjectLambdaEvent event, HttpRequest.Builder presignedRequest) { var header = event.getUserRequest().getHeaders().entrySet().stream() .filter(e -> e.getKey().toLowerCase(Locale.ROOT).equals("range")) .findFirst(); // Add check in the query string itself. header.ifPresent(entry -> presignedRequest.header(entry.getKey(), entry.getValue())); return presignedRequest; } ``` # Using AWS built Lambda functions **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). AWS provides some prebuilt AWS Lambda functions that you can use with Amazon S3 Object Lambda to detect and redact personally identifiable information (PII) and decompress S3 objects. These Lambda functions are available in the AWS Serverless Application Repository. You can select these functions through the AWS Management Console when you create your Object Lambda Access Point. For more information about how to deploy serverless applications from the AWS Serverless Application Repository, see [Deploying Applications](https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverlessrepo-consuming-applications.html) in the *AWS Serverless Application Repository Developer Guide*. **Note** The following examples can be used only with `GetObject` requests. ## Example 1: PII access control This Lambda function uses Amazon Comprehend, a natural language processing (NLP) service that uses machine learning to find insights and relationships in text. This function automatically detects personally identifiable information (PII), such as names, addresses, dates, credit card numbers, and social security numbers in documents in your Amazon S3 bucket. If you have documents in your bucket that include PII, you can configure the PII Access Control function to detect these PII entity types and restrict access to unauthorized users. To get started, deploy the following Lambda function in your account and add the Amazon Resource Name (ARN) for the function to your Object Lambda Access Point configuration. The following is an example ARN for this function: ``` arn:aws:serverlessrepo:us-east-1:111122223333:applications/ComprehendPiiAccessControlS3ObjectLambda ``` You can add or view this function on the AWS Management Console by using the following AWS Serverless Application Repository link: [ComprehendPiiAccessControlS3ObjectLambda](https://console.aws.amazon.com/lambda/home#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:839782855223:applications/ComprehendPiiAccessControlS3ObjectLambda). To view this function on GitHub, see [Amazon Comprehend S3 Object Lambda](https://github.com/aws-samples/amazon-comprehend-s3-object-lambdas). ## Example 2: PII redaction This Lambda function uses Amazon Comprehend, a natural language processing (NLP) service that uses machine learning to find insights and relationships in text. This function automatically redacts personally identifiable information (PII), such as names, addresses, dates, credit card numbers, and social security numbers from documents in your Amazon S3 bucket. If you have documents in your bucket that include information such as credit card numbers or bank account information, you can configure the PII Redaction S3 Object Lambda function to detect PII and then return a copy of these documents in which PII entity types are redacted. To get started, deploy the following Lambda function in your account and add the ARN for the function to your Object Lambda Access Point configuration. The following is an example ARN for this function: ``` arn:aws:serverlessrepo:us-east-1:111122223333::applications/ComprehendPiiRedactionS3ObjectLambda ``` You can add or view this function on the AWS Management Console by using the following AWS Serverless Application Repository link: [ComprehendPiiRedactionS3ObjectLambda](https://console.aws.amazon.com/lambda/home#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:839782855223:applications/ComprehendPiiRedactionS3ObjectLambda). To view this function on GitHub, see [Amazon Comprehend S3 Object Lambda](https://github.com/aws-samples/amazon-comprehend-s3-object-lambdas). To learn about complete end-to-end procedures for some S3 Object Lambda tasks in PII redaction, see [Tutorial: Detecting and redacting PII data with S3 Object Lambda and Amazon Comprehend](tutorial-s3-object-lambda-redact-pii.md). ## Example 3: Decompression The Lambda function `S3ObjectLambdaDecompression` can decompress objects that are stored in Amazon S3 in one of six compressed file formats: `bzip2`, `gzip`, `snappy`, `zlib`, `zstandard`, and `ZIP`. To get started, deploy the following Lambda function in your account and add the ARN for the function to your Object Lambda Access Point configuration. The following is an example ARN for this function: ``` arn:aws:serverlessrepo:us-east-1:111122223333::applications/S3ObjectLambdaDecompression ``` You can add or view this function on the AWS Management Console by using the following AWS Serverless Application Repository link: [S3ObjectLambdaDecompression](https://eu-west-1.console.aws.amazon.com/lambda/home?region=eu-west-1#/create/app?applicationId=arn:aws:serverlessrepo:eu-west-1:123065155563:applications/S3ObjectLambdaDecompression). To view this function on GitHub, see [S3 Object Lambda Decompression](https://github.com/aws-samples/amazon-s3-object-lambda-decompression). # Best practices and guidelines for S3 Object Lambda **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). When using S3 Object Lambda, follow these best practices and guidelines to optimize operations and performance. **Topics** + [ ## Working with S3 Object Lambda ](#olap-working-with) + [ ## AWS services used in connection with S3 Object Lambda ](#olap-services) + [ ## `Range` and `partNumber` headers ](#olap-managing-range-part) + [ ## Transforming the `expiry-date` ](#olap-console-download) + [ ## Working with the AWS CLI and AWS SDKs ](#olap-cli-sdk) ## Working with S3 Object Lambda S3 Object Lambda supports processing only `GET`, `LIST`, and `HEAD` requests. Any other requests don't invoke AWS Lambda and instead return standard, non-transformed API responses. You can create a maximum of 1,000 Object Lambda Access Points per AWS account per Region. The AWS Lambda function that you use must be in the same AWS account and Region as the Object Lambda Access Point. S3 Object Lambda allows up to 60 seconds to stream a complete response to its caller. Your function is also subject to AWS Lambda default quotas. For more information, see [Lambda quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html) in the *AWS Lambda Developer Guide*. When S3 Object Lambda invokes your specified Lambda function, you are responsible for ensuring that any data that is overwritten or deleted from Amazon S3 by your specified Lambda function or application is intended and correct. You can use S3 Object Lambda only to perform operations on objects. You cannot use S3 Object Lambda to perform other Amazon S3 operations, such as modifying or deleting buckets. For a complete list of S3 operations that support access points, see [Access points compatibility with S3 operations](access-points-service-api-support.md#access-points-operations-support). In addition to this list, Object Lambda Access Points do not support the [https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html), [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) (as the source), and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html) API operations. ## AWS services used in connection with S3 Object Lambda S3 Object Lambda connects Amazon S3, AWS Lambda, and optionally, other AWS services of your choosing to deliver objects relevant to the requesting applications. All AWS services used with S3 Object Lambda are governed by their respective Service Level Agreements (SLAs). For example, if any AWS service does not meet its Service Commitment, you are eligible to receive a Service Credit, as documented in the service's SLA. ## `Range` and `partNumber` headers When working with large objects, you can use the `Range` HTTP header to download a specified byte-range from an object. When you use the `Range` header, your request fetches only the specified portion of the object. You can also use the `partNumber` header to perform a ranged request for the specified part from the object. For more information, see [Working with Range and partNumber headers](range-get-olap.md). ## Transforming the `expiry-date` You can open or download transformed objects from your Object Lambda Access Point on the AWS Management Console. These objects must be non-expired. If your Lambda function transforms the `expiry-date` of your objects, you might see expired objects that cannot be opened or downloaded. This behavior applies only to S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive restored objects. ## Working with the AWS CLI and AWS SDKs AWS Command Line Interface (AWS CLI) S3 subcommands (`cp`, `mv`, and `sync`) and the use of the AWS SDK for Java `TransferManager` class are not supported for use with S3 Object Lambda. # S3 Object Lambda tutorials **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). The following tutorials present complete end-to-end procedures for some S3 Object Lambda tasks. With S3 Object Lambda, you can add your own code to process data retrieved from S3 before returning it to an application. Each of the following tutorials will modify data as it is retrieved from Amazon S3, without changing the existing object or maintaining multiple copies of the data. The first tutorial will walk through how to add an AWS Lambda function to a S3 GET request to modify an object retrieved from S3. The second tutorial demonstrates how to use a prebuilt Lambda function powered by Amazon Comprehend to protect personally identifiable information (PII) retrieved from S3 before returning it to an application. The third tutorial uses S3 Object Lambda to add a watermark to an image as it is retrieved from Amazon S3. + [Tutorial: Transforming data for your application with S3 Object Lambda](tutorial-s3-object-lambda-uppercase.md) + [Tutorial: Detecting and redacting PII data with S3 Object Lambda and Amazon Comprehend](tutorial-s3-object-lambda-redact-pii.md) + [Tutorial: Using S3 Object Lambda to dynamically watermark images as they are retrieved](https://aws.amazon.com/getting-started/hands-on/amazon-s3-object-lambda-to-dynamically-watermark-images/?ref=docs_gateway/amazons3/olap-tutorials.html) # Tutorial: Transforming data for your application with S3 Object Lambda **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). When you store data in Amazon S3, you can easily share it for use by multiple applications. However, each application might have unique data format requirements, and might need modification or processing of your data for a specific use case. For example, a dataset created by an ecommerce application might include personally identifiable information (PII). When the same data is processed for analytics, this PII is not needed and should be redacted. However, if the same dataset is used for a marketing campaign, you might need to enrich the data with additional details, such as information from the customer loyalty database. With [S3 Object Lambda](https://aws.amazon.com/s3/features/object-lambda), you can add your own code to process data retrieved from S3 before returning it to an application. Specifically, you can configure an AWS Lambda function and attach it to an S3 Object Lambda Access Point. When an application sends [standard S3 GET requests](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) through the S3 Object Lambda Access Point, the specified Lambda function is invoked to process any data retrieved from the underlying data source through the supporting S3 access point. Then, the S3 Object Lambda Access Point returns the transformed result back to the application. You can author and execute your own custom Lambda functions, tailoring the S3 Object Lambda data transformation to your specific use case, all with no changes required to your applications. ![\[This is an S3 Object Lambda workflow diagram.\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/images/ol-example-image-global.png) **Objective** In this tutorial, you learn how to add custom code to standard S3 GET requests to modify the requested object retrieved from S3 so that the object suit the needs of the requesting client or application. Specifically, you learn how to transform all the text in the original object stored in an S3 bucket to uppercase through S3 Object Lambda. **Note** This tutorial uses Python code to transform the data, for examples using other AWS SDKs see [Transform data for your application with S3 Object Lambda ](https://docs.aws.amazon.com/code-library/latest/ug/lambda_example_cross_ServerlessS3DataTransformation_section.html) in the AWS SDK Code Examples Library. **Topics** + [ ## Prerequisites ](#ol-upper-prerequisites) + [ ## Step 1: Create an S3 bucket ](#ol-upper-step1) + [ ## Step 2: Upload a file to the S3 bucket ](#ol-upper-step2) + [ ## Step 3: Create an S3 access point ](#ol-upper-step3) + [ ## Step 4: Create a Lambda function ](#ol-upper-step4) + [ ## Step 5: Configure an IAM policy for your Lambda function's execution role ](#ol-upper-step5) + [ ## Step 6: Create an S3 Object Lambda Access Point ](#ol-upper-step6) + [ ## Step 7: View the transformed data ](#ol-upper-step7) + [ ## Step 8: Clean up ](#ol-upper-step8) + [ ## Next steps ](#ol-upper-next-steps) ## Prerequisites Before you start this tutorial, you must have an AWS account that you can sign in to as an AWS Identity and Access Management (IAM) user with correct permissions. You also must install Python version 3.8 or later. **Topics** + [ ### Create an IAM user with permissions in your AWS account (console) ](#ol-upper-prerequisites-account) + [ ### Install Python 3.8 or later on your local machine ](#ol-upper-prerequisites-python) ### Create an IAM user with permissions in your AWS account (console) You can create an IAM user for the tutorial. To complete this tutorial, your IAM user must attach the following IAM policies to access relevant AWS resources and perform specific actions. For more information about how to create an IAM user, see [Creating IAM users (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console) in the *IAM User Guide*. Your IAM user requires the following policies: + [AmazonS3FullAccess](https://console.aws.amazon.com/iam/home?#/policies/arn:aws:iam::aws:policy/AmazonS3FullAccess$jsonEditor) – Grants permissions to all Amazon S3 actions, including permissions to create and use an Object Lambda Access Point. + [AWSLambda\$1FullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSLambda_FullAccess$jsonEditor) – Grants permissions to all Lambda actions. + [IAMFullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/IAMFullAccess$jsonEditor) – Grants permissions to all IAM actions. + [IAMAccessAnalyzerReadOnlyAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/IAMAccessAnalyzerReadOnlyAccess$jsonEditor) – Grants permissions to read all access information provided by IAM Access Analyzer. + [CloudWatchLogsFullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/CloudWatchLogsFullAccess$jsonEditor) – Grants full access to CloudWatch Logs. **Note** For simplicity, this tutorial creates and uses an IAM user. After completing this tutorial, remember to [Delete the IAM user](#ol-upper-step8-delete-user). For production use, we recommend that you follow the [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the *IAM User Guide*. A best practice requires human users to use federation with an identity provider to access AWS with temporary credentials. Another best practice is to require workloads to use temporary credentials with IAM roles to access AWS. To learn about using AWS IAM Identity Center to create users with temporary credentials, see [Getting started](https://docs.aws.amazon.com/singlesignon/latest/userguide/getting-started.html) in the *AWS IAM Identity Center User Guide*. This tutorial also uses full-access AWS managed policies. For production use, we recommend that you instead grant only the minimum permissions necessary for your use case, in accordance with [security best practices](security-best-practices.md#security-best-practices-prevent). ### Install Python 3.8 or later on your local machine Use the following procedure to install Python 3.8 or later on your local machine. For more installation instructions, see the [Downloading Python](https://wiki.python.org/moin/BeginnersGuide/Download) page in the *Python Beginners Guide*. 1. Open your local terminal or shell and run the following command to determine whether Python is already installed, and if so, which version is installed. ``` python --version ``` 1. If you don't have Python 3.8 or later, download the [official installer](https://www.python.org/downloads/) of Python 3.8 or later that's suitable for your local machine. 1. Run the installer by double-clicking the downloaded file, and follow the steps to complete the installation. For **Windows users**, choose **Add Python 3.X to PATH** in the installation wizard before choosing **Install Now**. 1. Restart your terminal by closing and reopening it. 1. Run the following command to verify that Python 3.8 or later is installed correctly. For **macOS users**, run this command: ``` python3 --version ``` For **Windows users**, run this command: ``` python --version ``` 1. Run the following command to verify that the pip3 package manager is installed. If you see a pip version number and python 3.8 or later in the command response, that means the pip3 package manager is installed successfully. ``` pip --version ``` ## Step 1: Create an S3 bucket Create a bucket to store the original data that you plan to transform. **Note** Access points may be attached for another data source, such as an Amazon FSx for OpenZFS volume, however this tutorial uses a supporting access point attached to an S3 bucket. **To create a bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. Choose **Create bucket**. The **Create bucket** page opens. 1. For **Bucket name**, enter a name (for example, **tutorial-bucket**) for your bucket. For more information about naming buckets in Amazon S3, see [General purpose bucket naming rules](bucketnamingrules.md). 1. For **Region**, choose the AWS Region where you want the bucket to reside. For more information about the bucket Region, see [General purpose buckets overview](UsingBucket.md). 1. For **Block Public Access settings for this bucket**, keep the default settings (**Block *all *public access** is enabled). We recommend that you keep all Block Public Access settings enabled unless you need to turn off one or more of them for your use case. For more information about blocking public access, see [Blocking public access to your Amazon S3 storage](access-control-block-public-access.md). 1. For the remaining settings, keep the defaults. (Optional) If you want to configure additional bucket settings for your specific use case, see [Creating a general purpose bucket](create-bucket-overview.md). 1. Choose **Create bucket**. ## Step 2: Upload a file to the S3 bucket Upload a text file to the S3 bucket. This text file contains the original data that you will transform to uppercase later in this tutorial. For example, you can upload a `tutorial.txt` file that contains the following text: ``` Amazon S3 Object Lambda Tutorial: You can add your own code to process data retrieved from S3 before returning it to an application. ``` **To upload a file to a bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the name of the bucket that you created in [Step 1](#ol-upper-step1) (for example, **tutorial-bucket**) to upload your file to. 1. On the **Objects** tab for your bucket, choose **Upload**. 1. On the **Upload** page, under **Files and folders**, choose **Add files**. 1. Choose a file to upload, and then choose **Open**. For example, you can upload the `tutorial.txt` file example mentioned earlier. 1. Choose **Upload**. ## Step 3: Create an S3 access point To use an S3 Object Lambda Access Point to access and transform the original data, you must create an S3 access point and associate it with the S3 bucket that you created in [Step 1](#ol-upper-step1). The access point must be in the same AWS Region as the objects that you want to transform. Later in this tutorial, you'll use this access point as a supporting access point for your Object Lambda Access Point. **To create an access point** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Access Points**. 1. On the **Access Points** page, choose **Create access point**. 1. In the **Access point name** field, enter the name (for example, **tutorial-access-point**) for the access point. For more information about naming access points, see [Naming rules for access points](access-points-restrictions-limitations-naming-rules.md#access-points-names). 1. In the **Data source** field, enter the name of the bucket that you created in [Step 1](#ol-upper-step1) (for example, **tutorial-bucket**). S3 attaches the access point to this bucket. (Optional) You can choose **Browse S3** to browse and search the buckets in your account. If you choose **Browse S3**, choose the desired bucket, and then choose **Choose path** to populate the **Bucket name** field with that bucket's name. 1. For **Network origin**, choose **Internet**. For more information about network origins for access points, see [Creating access points restricted to a virtual private cloud](access-points-vpc.md). 1. By default, all Block Public Access settings are turned on for your access point. We recommend that you keep **Block *all* public access** enabled. For more information, see [Managing public access to access points for general purpose buckets](access-points-bpa-settings.md). 1. For all other access point settings, keep the default settings. (Optional) You can modify the access point settings to support your use case. For this tutorial, we recommend keeping the default settings. (Optional) If you need to manage access to your access point, you can specify an access point policy. For more information, see [Policy examples for access points](access-points-policies.md#access-points-policy-examples). 1. Choose **Create access point**. ## Step 4: Create a Lambda function To transform original data, create a Lambda function for use with your S3 Object Lambda Access Point. **Topics** + [ ### Write Lambda function code and create a deployment package with a virtual environment ](#ol-upper-step4-write-lambda) + [ ### Create a Lambda function with an execution role (console) ](#ol-upper-step4-create-function) + [ ### Deploy your Lambda function code with .zip file archives and configure the Lambda function (console) ](#ol-upper-step4-deploy-function) ### Write Lambda function code and create a deployment package with a virtual environment 1. On your local machine, create a folder with the folder name `object-lambda` for the virtual environment to use later in this tutorial. 1. In the `object-lambda` folder, create a file with a Lambda function that changes all text in the original object to uppercase. For example, you can use the following function written in Python. Save this function in a file named `transform.py`. ``` import boto3 import requests from botocore.config import Config # This function capitalizes all text in the original object def lambda_handler(event, context): object_context = event["getObjectContext"] # Get the presigned URL to fetch the requested original object # from S3 s3_url = object_context["inputS3Url"] # Extract the route and request token from the input context request_route = object_context["outputRoute"] request_token = object_context["outputToken"] # Get the original S3 object using the presigned URL response = requests.get(s3_url) original_object = response.content.decode("utf-8") # Transform all text in the original object to uppercase # You can replace it with your custom code based on your use case transformed_object = original_object.upper() # Write object back to S3 Object Lambda s3 = boto3.client('s3', config=Config(signature_version='s3v4')) # The WriteGetObjectResponse API sends the transformed data # back to S3 Object Lambda and then to the user s3.write_get_object_response( Body=transformed_object, RequestRoute=request_route, RequestToken=request_token) # Exit the Lambda function: return the status code return {'status_code': 200} ``` **Note** The preceding example Lambda function loads the entire requested object into memory before transforming it and returning it to the client. Alternatively, you can stream the object from S3 to avoid loading the entire object into memory. This approach can be useful when working with large objects. For more information about streaming responses with Object Lambda Access Points, see the streaming examples in [Working with `GetObject` requests in Lambda](olap-writing-lambda.md#olap-getobject-response). When you're writing a Lambda function for use with an S3 Object Lambda Access Point, the function is based on the input event context that S3 Object Lambda provides to the Lambda function. The event context provides information about the request being made in the event passed from S3 Object Lambda to Lambda. It contains the parameters that you use to create the Lambda function. The fields used to create the preceding Lambda function are as follows: The field of `getObjectContext` means the input and output details for connections to Amazon S3 and S3 Object Lambda. It has the following fields: + `inputS3Url` – A presigned URL that the Lambda function can use to download the original object from the supporting access point. By using a presigned URL, the Lambda function doesn't need to have Amazon S3 read permissions to retrieve the original object and can only access the object processed by each invocation. + `outputRoute` – A routing token that is added to the S3 Object Lambda URL when the Lambda function calls `WriteGetObjectResponse` to send back the transformed object. + `outputToken` – A token used by S3 Object Lambda to match the `WriteGetObjectResponse` call with the original caller when sending back the transformed object. For more information about all the fields in the event context, see [Event context format and usage](olap-event-context.md) and [Writing Lambda functions for S3 Object Lambda Access Points](olap-writing-lambda.md). 1. In your local terminal, enter the following command to install the `virtualenv` package: ``` python -m pip install virtualenv ``` 1. In your local terminal, open the `object-lambda` folder that you created earlier, and then enter the following command to create and initialize a virtual environment called `venv`. ``` python -m virtualenv venv ``` 1. To activate the virtual environment, enter the following command to execute the `activate` file from the environment's folder: For **macOS users**, run this command: ``` source venv/bin/activate ``` For **Windows users**, run this command: ``` .\venv\Scripts\activate ``` Now, your command prompt changes to show **(venv)**, indicating that the virtual environment is active. 1. To install the required libraries, run the following commands line by line in the `venv` virtual environment. These commands install updated versions of the dependencies of your `lambda_handler` Lambda function. These dependencies are the AWS SDK for Python (Boto3) and the requests module. ``` pip3 install boto3 ``` ``` pip3 install requests ``` 1. To deactivate the virtual environment, run the following command: ``` deactivate ``` 1. To create a deployment package with the installed libraries as a `.zip` file named `lambda.zip` at the root of the `object-lambda` directory, run the following commands line by line in your local terminal. **Tip** The following commands might need to be adjusted to work in your particular environment. For example, a library might appear in `site-packages` or `dist-packages`, and the first folder might be `lib` or `lib64`. Also, the `python` folder might be named with a different Python version. To locate a specific package, use the `pip show` command. For **macOS users**, run these commands: ``` cd venv/lib/python3.8/site-packages ``` ``` zip -r ../../../../lambda.zip . ``` For **Windows users**, run these commands: ``` cd .\venv\Lib\site-packages\ ``` ``` powershell Compress-Archive * ../../../lambda.zip ``` The last command saves the deployment package to the root of the `object-lambda` directory. 1. Add the function code file `transform.py` to the root of your deployment package. For **macOS users**, run these commands: ``` cd ../../../../ ``` ``` zip -g lambda.zip transform.py ``` For **Windows users**, run these commands: ``` cd ..\..\..\ ``` ``` powershell Compress-Archive -update transform.py lambda.zip ``` After you complete this step, you should have the following directory structure: ``` lambda.zip$ │ transform.py │ __pycache__ | boto3/ │ certifi/ │ pip/ │ requests/ ... ``` ### Create a Lambda function with an execution role (console) 1. Sign in to the AWS Management Console and open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. In the left navigation pane, choose **Functions**. 1. Choose **Create function**. 1. Choose **Author from scratch**. 1. Under **Basic information**, do the following: 1. For **Function name**, enter **tutorial-object-lambda-function**. 1. For **Runtime**, choose **Python 3.8** or a later version. 1. Expand the **Change default execution role** section. Under **Execution role**, choose **Create a new role with basic Lambda permissions**. In [Step 5](#ol-upper-step5) later in this tutorial, you attach the **AmazonS3ObjectLambdaExecutionRolePolicy** to this Lambda function's execution role. 1. Keep the remaining settings set to the defaults. 1. Choose **Create function**. ### Deploy your Lambda function code with .zip file archives and configure the Lambda function (console) 1. In the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/), choose **Functions** in the left navigation pane. 1. Choose the Lambda function that you created earlier (for example, **tutorial-object-lambda-function**). 1. On the Lambda function's details page, choose the **Code** tab. In the **Code Source** section, choose **Upload from** and then **.zip file**. 1. Choose **Upload** to select your local `.zip` file. 1. Choose the `lambda.zip` file that you created earlier, and then choose **Open**. 1. Choose **Save**. 1. In the **Runtime settings** section, choose **Edit**. 1. On the **Edit runtime settings** page, confirm that **Runtime** is set to **Python 3.8** or a later version. 1. To tell the Lambda runtime which handler method in your Lambda function code to invoke, enter **transform.lambda\$1handler** for **Handler**. When you configure a function in Python, the value of the handler setting is the file name and the name of the handler module, separated by a dot. For example, `transform.lambda_handler` calls the `lambda_handler` method defined in the `transform.py` file. 1. Choose **Save**. 1. (Optional) On your Lambda function's details page, choose the **Configuration** tab. In the left navigation pane, choose **General configuration**, then choose **Edit**. In the **Timeout** field, enter **1** min **0** sec. Keep the remaining settings set to the defaults, and choose **Save**. **Timeout** is the amount of time that Lambda allows a function to run for an invocation before stopping it. The default is 3 seconds. The maximum duration for a Lambda function used by S3 Object Lambda is 60 seconds. Pricing is based on the amount of memory configured and the amount of time that your code runs. ## Step 5: Configure an IAM policy for your Lambda function's execution role To enable your Lambda function to provide customized data and response headers to the `GetObject` caller, your Lambda function's execution role must have IAM permissions to call the `WriteGetObjectResponse` API. **To attach an IAM policy to your Lambda function role** 1. In the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/), choose **Functions** in the left navigation pane. 1. Choose the function that you created in [Step 4](#ol-upper-step4) (for example, **tutorial-object-lambda-function**). 1. On your Lambda function's details page, choose the **Configuration** tab, and then choose **Permissions** in the left navigation pane. 1. Under **Execution role**, choose the link of the **Role name**. The IAM console opens. 1. On the IAM console's **Summary** page for your Lambda function's execution role, choose the **Permissions** tab. Then, from the **Add Permissions** menu, choose **Attach policies**. 1. On the **Attach Permissions** page, enter **AmazonS3ObjectLambdaExecutionRolePolicy** in the search box to filter the list of policies. Select the check box next to the name of the **AmazonS3ObjectLambdaExecutionRolePolicy** policy. 1. Choose **Attach policies**. ## Step 6: Create an S3 Object Lambda Access Point An S3 Object Lambda Access Point provides the flexibility to invoke a Lambda function directly from an S3 GET request so that the function can process data retrieved from an S3 access point. When creating and configuring an S3 Object Lambda Access Point, you must specify the Lambda function to invoke and provide the event context in JSON format as custom parameters for Lambda to use. **To create an S3 Object Lambda Access Point** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose **Create Object Lambda Access Point**. 1. For **Object Lambda Access Point name**, enter the name that you want to use for the Object Lambda Access Point (for example, **tutorial-object-lambda-accesspoint**). 1. For **Supporting Access Point**, enter or browse to the standard access point that you created in [Step 3](#ol-upper-step3) (for example, **tutorial-access-point**), and then choose **Choose supporting Access Point**. 1. For **S3 APIs**, to retrieve objects from the S3 bucket for Lambda function to process, select **GetObject**. 1. For **Invoke Lambda function**, you can choose either of the following two options for this tutorial. + Choose **Choose from functions in your account**, and then choose the Lambda function that you created in [Step 4](#ol-upper-step4) (for example, **tutorial-object-lambda-function**) from the **Lambda function** dropdown list. + Choose **Enter ARN**, and then enter the Amazon Resource Name (ARN) of the Lambda function that you created in [Step 4](#ol-upper-step4). 1. For **Lambda function version**, choose **\$1LATEST** (the latest version of the Lambda function that you created in [Step 4](#ol-upper-step4)). 1. (Optional) If you need your Lambda function to recognize and process GET requests with range and part number headers, select **Lambda function supports requests using range** and **Lambda function supports requests using part numbers**. Otherwise, clear these two check boxes. For more information about how to use range or part numbers with S3 Object Lambda, see [Working with Range and partNumber headers](range-get-olap.md). 1. (Optional) Under **Payload - *optional***, add JSON text to provide your Lambda function with additional information. A payload is optional JSON text that you can provide to your Lambda function as input for all invocations coming from a specific S3 Object Lambda Access Point. To customize the behaviors for multiple Object Lambda Access Points that invoke the same Lambda function, you can configure payloads with different parameters, thereby extending the flexibility of your Lambda function. For more information about payload, see [Event context format and usage](olap-event-context.md). 1. (Optional) For **Request metrics - *optional***, choose **Disable** or **Enable** to add Amazon S3 monitoring to your Object Lambda Access Point. Request metrics are billed at the standard Amazon CloudWatch rate. For more information, see [CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). 1. Under **Object Lambda Access Point policy - *optional***, keep the default setting. (Optional) You can set a resource policy. This resource policy grants the `GetObject` API permission to use the specified Object Lambda Access Point. 1. Keep the remaining settings set to the defaults, and choose **Create Object Lambda Access Point**. ## Step 7: View the transformed data Now, S3 Object Lambda is ready to transform your data for your use case. In this tutorial, S3 Object Lambda transforms all the text in your object to uppercase. **Topics** + [ ### View the transformed data in your S3 Object Lambda Access Point ](#ol-upper-step7-check-data) + [ ### Run a Python script to print the original and transformed data ](#ol-upper-step7-python-print) ### View the transformed data in your S3 Object Lambda Access Point When you request to retrieve a file through your S3 Object Lambda Access Point, you make a `GetObject` API call to S3 Object Lambda. S3 Object Lambda invokes the Lambda function to transform your data, and then returns the transformed data as the response to the standard S3 `GetObject` API call. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose the S3 Object Lambda Access Point that you created in [Step 6](#ol-upper-step6) (for example, **tutorial-object-lambda-accesspoint**). 1. On the **Objects** tab of your S3 Object Lambda Access Point, select the file that has the same name (for example, `tutorial.txt`) as the one that you uploaded to the S3 bucket in [Step 2](#ol-upper-step2). This file should contain all the transformed data. 1. To view the transformed data, choose **Open** or **Download**. ### Run a Python script to print the original and transformed data You can use S3 Object Lambda with your existing applications. To do so, update your application configuration to use the new S3 Object Lambda Access Point ARN that you created in [Step 6](#ol-upper-step6) to retrieve data from S3. The following example Python script prints both the original data from the S3 bucket and the transformed data from the S3 Object Lambda Access Point. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose the radio button to the left of the S3 Object Lambda Access Point that you created in [Step 6](#ol-upper-step6) (for example, **tutorial-object-lambda-accesspoint**). 1. Choose **Copy ARN**. 1. Save the ARN for use later. 1. Write a Python script on your local machine to print both the original data (for example, `tutorial.txt`) from your S3 Bucket and the transformed data (for example, `tutorial.txt`) from your S3 Object Lambda Access Point). You can use the following example script. ``` import boto3 from botocore.config import Config s3 = boto3.client('s3', config=Config(signature_version='s3v4')) def getObject(bucket, key): objectBody = s3.get_object(Bucket = bucket, Key = key) print(objectBody["Body"].read().decode("utf-8")) print("\n") print('Original object from the S3 bucket:') # Replace the two input parameters of getObject() below with # the S3 bucket name that you created in Step 1 and # the name of the file that you uploaded to the S3 bucket in Step 2 getObject("tutorial-bucket", "tutorial.txt") print('Object transformed by S3 Object Lambda:') # Replace the two input parameters of getObject() below with # the ARN of your S3 Object Lambda Access Point that you saved earlier and # the name of the file with the transformed data (which in this case is # the same as the name of the file that you uploaded to the S3 bucket # in Step 2) getObject("arn:aws:s3-object-lambda:us-west-2:111122223333:accesspoint/tutorial-object-lambda-accesspoint", "tutorial.txt") ``` 1. Save your Python script with a custom name (for example, `tutorial_print.py` ) in the folder (for example, `object-lambda`) that you created in [Step 4](#ol-upper-step4) on your local machine. 1. In your local terminal, run the following command from the root of the directory (for example, `object-lambda`) that you created in [Step 4](#ol-upper-step4). ``` python3 tutorial_print.py ``` You should see both the original data and the transformed data (all text as uppercase) through the terminal. For example, you should see something like the following text. ``` Original object from the S3 bucket: Amazon S3 Object Lambda Tutorial: You can add your own code to process data retrieved from S3 before returning it to an application. Object transformed by S3 Object Lambda: AMAZON S3 OBJECT LAMBDA TUTORIAL: YOU CAN ADD YOUR OWN CODE TO PROCESS DATA RETRIEVED FROM S3 BEFORE RETURNING IT TO AN APPLICATION. ``` ## Step 8: Clean up If you transformed your data through S3 Object Lambda only as a learning exercise, delete the AWS resources that you allocated so that you no longer accrue charges. **Topics** + [ ### Delete the Object Lambda Access Point ](#ol-upper-step8-delete-olap) + [ ### Delete the S3 access point ](#ol-upper-step8-delete-ap) + [ ### Delete the execution role for your Lambda function ](#ol-upper-step8-delete-lambda-role) + [ ### Delete the Lambda function ](#ol-upper-step8-delete-lambda-function) + [ ### Delete the CloudWatch log group ](#ol-upper-step8-delete-cloudwatch) + [ ### Delete the original file in the S3 source bucket ](#ol-upper-step8-delete-file) + [ ### Delete the S3 source bucket ](#ol-upper-step8-delete-bucket) + [ ### Delete the IAM user ](#ol-upper-step8-delete-user) ### Delete the Object Lambda Access Point 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose the radio button to the left of the S3 Object Lambda Access Point that you created in [Step 6](#ol-upper-step6) (for example, **tutorial-object-lambda-accesspoint**). 1. Choose **Delete**. 1. Confirm that you want to delete your Object Lambda Access Point by entering its name in the text field that appears, and then choose **Delete**. ### Delete the S3 access point 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Access Points**. 1. Navigate to the access point that you created in [Step 3](#ol-upper-step3) (for example, **tutorial-access-point**), and choose the radio button next to the name of the access point. 1. Choose **Delete**. 1. Confirm that you want to delete your access point by entering its name in the text field that appears, and then choose **Delete**. ### Delete the execution role for your Lambda function 1. Sign in to the AWS Management Console and open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. In the left navigation pane, choose **Functions**. 1. Choose the function that you created in [Step 4](#ol-upper-step4) (for example, **tutorial-object-lambda-function**). 1. On your Lambda function's details page, choose the **Configuration** tab, and then choose **Permissions** in the left navigation pane. 1. Under **Execution role**, choose the link of the **Role name**. The IAM console opens. 1. On the IAM console's **Summary** page of your Lambda function's execution role, choose **Delete role**. 1. In the **Delete role** dialog box, choose **Yes, delete**. ### Delete the Lambda function 1. In the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/), choose **Functions** in the left navigation pane. 1. Select the check box to the left of the name of the function that you created in [Step 4](#ol-upper-step4) (for example, **tutorial-object-lambda-function**). 1. Choose **Actions**, and then choose **Delete**. 1. In the **Delete function** dialog box, choose **Delete**. ### Delete the CloudWatch log group 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the left navigation pane, choose **Log groups**. 1. Find the log group whose name ends with the Lambda function that you created in [Step 4](#ol-upper-step4) (for example, **tutorial-object-lambda-function**). 1. Select the check box to the left of the name of the log group. 1. Choose **Actions**, and then choose **Delete log group(s)**. 1. In the **Delete log group(s)** dialog box, choose **Delete**. ### Delete the original file in the S3 source bucket 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Bucket name** list, choose the name of the bucket that you uploaded the original file to in [Step 2](#ol-upper-step2) (for example, **tutorial-bucket**). 1. Select the check box to the left of the name of the object that you want to delete (for example, `tutorial.txt`). 1. Choose **Delete**. 1. On the **Delete objects** page, in the **Permanently delete objects?** section, confirm that you want to delete this object by entering **permanently delete** in the text box. 1. Choose **Delete objects**. ### Delete the S3 source bucket 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the radio button next to the name of the bucket that you created in [Step 1](#ol-upper-step1) (for example, **tutorial-bucket**). 1. Choose **Delete**. 1. On the **Delete bucket** page, confirm that you want to delete the bucket by entering the bucket name in the text field, and then choose **Delete bucket**. ### Delete the IAM user 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Users**, and then select the check box next to the user name that you want to delete. 1. At the top of the page, choose **Delete**. 1. In the **Delete *user name*?** dialog box, enter the user name in the text input field to confirm the deletion of the user. Choose **Delete**. ## Next steps After completing this tutorial, you can customize the Lambda function for your use case to modify the data returned by standard S3 GET requests. The following is a list of common use cases for S3 Object Lambda: + Masking sensitive data for security and compliance. For more information, see [Tutorial: Detecting and redacting PII data with S3 Object Lambda and Amazon Comprehend](tutorial-s3-object-lambda-redact-pii.md). + Filtering certain rows of data to deliver specific information. + Augmenting data with information from other services or databases. + Converting across data formats, such as converting XML to JSON for application compatibility. + Compressing or decompressing files as they are being downloaded. + Resizing and watermarking images. For more information, see [Tutorial: Using S3 Object Lambda to dynamically watermark images as they are retrieved](https://aws.amazon.com/getting-started/hands-on/amazon-s3-object-lambda-to-dynamically-watermark-images/?ref=docs_gateway/amazons3/tutorial-s3-object-lambda-uppercase.html). + Implementing custom authorization rules to access data. For more information about S3 Object Lambda, see [Transforming objects with S3 Object Lambda](transforming-objects.md). # Tutorial: Detecting and redacting PII data with S3 Object Lambda and Amazon Comprehend **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). When you're using Amazon S3 for shared datasets for multiple applications and users to access, it's important to restrict privileged information, such as personally identifiable information (PII), to only authorized entities. For example, when a marketing application uses some data containing PII, it might need to first mask PII data to meet data privacy requirements. Also, when an analytics application uses a production order inventory dataset, it might need to first redact customer credit card information to prevent unintended data leakage. With [S3 Object Lambda](https://aws.amazon.com/s3/features/object-lambda) and a prebuilt AWS Lambda function powered by Amazon Comprehend, you can protect PII data retrieved from S3 before returning it to an application. Specifically, you can use the prebuilt [Lambda function](https://aws.amazon.com/lambda/) as a redacting function and attach it to an S3 Object Lambda Access Point. When an application (for example, an analytics application) sends [standard S3 GET requests](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html), these requests made through the S3 Object Lambda Access Point invoke the prebuilt redacting Lambda function to detect and redact PII data retrieved from an underlying data source through a supporting S3 access point. Then, the S3 Object Lambda Access Point returns the redacted result back to the application. ![\[This is an S3 Object Lambda workflow diagram.\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/images/ol-comprehend-image-global.png) In the process, the prebuilt Lambda function uses [Amazon Comprehend](https://aws.amazon.com/comprehend/), a natural language processing (NLP) service, to capture variations in how PII is represented, regardless of how PII exists in text (such as numerically or as a combination of words and numbers). Amazon Comprehend can even use context in the text to understand if a 4-digit number is a PIN, the last four numbers of a Social Security number (SSN), or a year. Amazon Comprehend processes any text file in UTF-8 format and can protect PII at scale without affecting accuracy. For more information, see [What is Amazon Comprehend?](https://docs.aws.amazon.com/comprehend/latest/dg/what-is.html) in the *Amazon Comprehend Developer Guide*. **Objective** In this tutorial, you learn how to use S3 Object Lambda with the prebuilt Lambda function `ComprehendPiiRedactionS3ObjectLambda`. This function uses Amazon Comprehend to detect PII entities. It then redacts these entities by replacing them with asterisks. By redacting PII, you conceal sensitive data, which can help with security and compliance. You also learn how to use and configure a prebuilt AWS Lambda function in the [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/) to work together with S3 Object Lambda for easy deployment. **Topics** + [ ## Prerequisites: Create an IAM user with permissions ](#ol-pii-prerequisites) + [ ## Step 1: Create an S3 bucket ](#ol-pii-step1) + [ ## Step 2: Upload a file to the S3 bucket ](#ol-pii-step2) + [ ## Step 3: Create an S3 access point ](#ol-pii-step3) + [ ## Step 4: Configure and deploy a prebuilt Lambda function ](#ol-pii-step4) + [ ## Step 5: Create an S3 Object Lambda Access Point ](#ol-pii-step5) + [ ## Step 6: Use the S3 Object Lambda Access Point to retrieve the redacted file ](#ol-pii-step6) + [ ## Step 7: Clean up ](#ol-pii-step7) + [ ## Next steps ](#ol-pii-next-steps) ## Prerequisites: Create an IAM user with permissions Before you start this tutorial, you must have an AWS account that you can sign in to as an AWS Identity and Access Management user (IAM user) with correct permissions. You can create an IAM user for the tutorial. To complete this tutorial, your IAM user must attach the following IAM policies to access relevant AWS resources and perform specific actions. **Note** For simplicity, this tutorial creates and uses an IAM user. After completing this tutorial, remember to [Delete the IAM user](#ol-pii-step8-delete-user). For production use, we recommend that you follow the [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the *IAM User Guide*. A best practice requires human users to use federation with an identity provider to access AWS with temporary credentials. Another best practice is to require workloads to use temporary credentials with IAM roles to access AWS. To learn about using AWS IAM Identity Center to create users with temporary credentials, see [Getting started](https://docs.aws.amazon.com/singlesignon/latest/userguide/getting-started.html) in the *AWS IAM Identity Center User Guide*. This tutorial also uses full-access policies. For production use, we recommend that you instead grant only the minimum permissions necessary for your use case, in accordance with [security best practices](security-best-practices.md#security-best-practices-prevent). Your IAM user requires the following AWS managed policies: + [AmazonS3FullAccess](https://console.aws.amazon.com/iam/home?#/policies/arn:aws:iam::aws:policy/AmazonS3FullAccess$jsonEditor) – Grants permissions to all Amazon S3 actions, including permissions to create and use an Object Lambda Access Point. + [AWSLambda\$1FullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSLambda_FullAccess$jsonEditor) – Grants permissions to all Lambda actions. + [AWSCloudFormationFullAccess](https://console.aws.amazon.com/iam/home?#/policies/arn:aws:iam::aws:policy/AWSCloudFormationFullAccess$serviceLevelSummary) – Grants permissions to all AWS CloudFormation actions. + [IAMFullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/IAMFullAccess$jsonEditor) – Grants permissions to all IAM actions. + [IAMAccessAnalyzerReadOnlyAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/IAMAccessAnalyzerReadOnlyAccess$jsonEditor) – Grants permissions to read all access information provided by IAM Access Analyzer. You can directly attach these existing policies when creating an IAM user. For more information about how to create an IAM user, see [Creating IAM users (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console) in the *IAM User Guide*. In addition, your IAM user requires a customer managed policy. To grant the IAM user permissions to all AWS Serverless Application Repository resources and actions, you must create an IAM policy and attach the policy to the IAM user. **To create and attach an IAM policy to your IAM user** 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policies**. 1. Choose **Create policy**. 1. On the **Visual editor** tab, for **Service**, choose **Choose a service**. Then, choose **Serverless Application Repository**. 1. For **Actions**, under **Manual actions**, select **All Serverless Application Repository actions (serverlessrepo:\$1)** for this tutorial. As a security best practice, you should allow permissions to only those actions and resources that a user needs, based on your use case. For more information, see [ Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the *IAM User Guide*. 1. For **Resources**, choose **All resources** for this tutorial. As a best practice, you should define permissions for only specific resources in specific accounts. Alternatively, you can grant least privilege using condition keys. For more information, see [ Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) in the *IAM User Guide*. 1. Choose **Next: Tags**. 1. Choose **Next: Review**. 1. On the **Review policy** page, enter a **Name** (for example, **tutorial-serverless-application-repository**) and a **Description** (optional) for the policy that you are creating. Review the policy summary to make sure that you have granted the intended permissions, and then choose **Create policy** to save your new policy. 1. In the left navigation pane, choose **Users**. Then, choose the IAM user for this tutorial. 1. On the **Summary** page of the chosen user, choose the **Permissions** tab, and then choose **Add permissions**. 1. Under **Grant permissions**, choose **Attach existing policies directly**. 1. Select the check box next to the policy that you just created (for example, **tutorial-serverless-application-repository**) and then choose **Next: Review**. 1. Under **Permissions summary**, review the summary to make sure that you attached the intended policy. Then, choose **Add permissions**. ## Step 1: Create an S3 bucket Create a bucket to store the original data that you plan to transform. **Note** Access points may be attached for another data source, such as an Amazon FSx for OpenZFS volume, however this tutorial uses a supporting access point attached to an S3 bucket. **To create a bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. Choose **Create bucket**. The **Create bucket** page opens. 1. For **Bucket name**, enter a name (for example, **tutorial-bucket**) for your bucket. For more information about naming buckets in Amazon S3, see [General purpose bucket naming rules](bucketnamingrules.md). 1. For **Region**, choose the AWS Region where you want the bucket to reside. For more information about the bucket Region, see [General purpose buckets overview](UsingBucket.md). 1. For **Block Public Access settings for this bucket**, keep the default settings (**Block *all *public access** is enabled). We recommend that you keep all Block Public Access settings enabled unless you need to turn off one or more of them for your use case. For more information about blocking public access, see [Blocking public access to your Amazon S3 storage](access-control-block-public-access.md). 1. For the remaining settings, keep the defaults. (Optional) If you want to configure additional bucket settings for your specific use case, see [Creating a general purpose bucket](create-bucket-overview.md). 1. Choose **Create bucket**. ## Step 2: Upload a file to the S3 bucket Upload a text file containing known PII data of various types, such as names, banking information, phone numbers, and SSNs, to the S3 bucket as the original data that you will redact PII from later in this tutorial. For example, you can upload following the `tutorial.txt` file. This is an example input file from Amazon Comprehend. ``` Hello Zhang Wei, I am John. Your AnyCompany Financial Services, LLC credit card account 1111-0000-1111-0008 has a minimum payment of $24.53 that is due by July 31st. Based on your autopay settings, we will withdraw your payment on the due date from your bank account number XXXXXX1111 with the routing number XXXXX0000. Your latest statement was mailed to 100 Main Street, Any City, WA 98121. After your payment is received, you will receive a confirmation text message at 206-555-0100. If you have questions about your bill, AnyCompany Customer Service is available by phone at 206-555-0199 or email at support@anycompany.com. ``` **To upload a file to a bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the name of the bucket that you created in [Step 1](#ol-pii-step1) (for example, **tutorial-bucket**) to upload your file to. 1. On the **Objects** tab for your bucket, choose **Upload**. 1. On the **Upload** page, under **Files and folders**, choose **Add files**. 1. Choose a file to upload, and then choose **Open**. For example, you can upload the `tutorial.txt` file example mentioned earlier. 1. Choose **Upload**. ## Step 3: Create an S3 access point To use an S3 Object Lambda Access Point to access and transform the original data, you must create an S3 access point and associate it with the S3 bucket that you created in [Step 1](#ol-pii-step1). The access point must be in the same AWS Region as the objects you want to transform. Later in this tutorial, you'll use this access point as a supporting access point for your Object Lambda Access Point. **To create an access point** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Access Points**. 1. On the **Access Points** page, choose **Create access point**. 1. In the **Access point name** field, enter the name (for example, **tutorial-pii-access-point**) for the access point. For more information about naming access points, see [Naming rules for access points](access-points-restrictions-limitations-naming-rules.md#access-points-names). 1. In the **Data source** field, enter the name of the bucket that you created in [Step 1](#ol-pii-step1) (for example, **tutorial-bucket**). S3 attaches the access point to this bucket. (Optional) You can choose **Browse S3** to browse and search the buckets in your account. If you choose **Browse S3**, choose the desired bucket, and then choose **Choose path** to populate the **Bucket name** field with that bucket's name. 1. For **Network origin**, choose **Internet**. For more information about network origins for access points, see [Creating access points restricted to a virtual private cloud](access-points-vpc.md). 1. By default, all block public access settings are turned on for your access point. We recommend that you keep **Block *all* public access** enabled. For more information, see [Managing public access to access points for general purpose buckets](access-points-bpa-settings.md). 1. For all other access point settings, keep the default settings. (Optional) You can modify the access point settings to support your use case. For this tutorial, we recommend keeping the default settings. (Optional) If you need to manage access to your access point, you can specify an access point policy. For more information, see [Policy examples for access points](access-points-policies.md#access-points-policy-examples). 1. Choose **Create access point**. ## Step 4: Configure and deploy a prebuilt Lambda function To redact PII data, configure and deploy the prebuilt AWS Lambda function `ComprehendPiiRedactionS3ObjectLambda` for use with your S3 Object Lambda Access Point. **To configure and deploy the Lambda function** 1. Sign in to the AWS Management Console and view the [https://console.aws.amazon.com/lambda/home#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:839782855223:applications/ComprehendPiiRedactionS3ObjectLambda](https://console.aws.amazon.com/lambda/home#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:839782855223:applications/ComprehendPiiRedactionS3ObjectLambda) function in the AWS Serverless Application Repository. 1. For **Application settings**, under **Application name**, keep the default value (`ComprehendPiiRedactionS3ObjectLambda`) for this tutorial. (Optional) You can enter the name that you want to give to this application. You might want to do this if you plan to configure multiple Lambda functions for different access needs for the same shared dataset. 1. For **MaskCharacter**, keep the default value (`*`). The mask character replaces each character in the redacted PII entity. 1. For **MaskMode**, keep the default value (**MASK**). The **MaskMode** value specifies whether the PII entity is redacted with the `MASK` character or the `PII_ENTITY_TYPE` value. 1. To redact the specified types of data, for **PiiEntityTypes**, keep the default value **ALL**. The **PiiEntityTypes** value specifies the PII entity types to be considered for redaction. For more information about the list of supported PII entity types, see [Detect Personally Identifiable Information (PII)](https://docs.aws.amazon.com/comprehend/latest/dg/how-pii.html) in the *Amazon Comprehend Developer Guide*. 1. Keep the remaining settings set to the defaults. (Optional) If you want to configure additional settings for your specific use case, see the **Readme file** section on the left side of the page. 1. Select the check box next to **I acknowledge that this app creates custom IAM roles**. 1. Choose **Deploy**. 1. On the new application's page, under **Resources**, choose the **Logical ID** of the Lambda function that you deployed to review the function on the Lambda function page. ## Step 5: Create an S3 Object Lambda Access Point An S3 Object Lambda Access Point provides the flexibility to invoke a Lambda function directly from an S3 GET request so that the function can redact PII data retrieved from an S3 access point. When creating and configuring an S3 Object Lambda Access Point, you must specify the redacting Lambda function to invoke and provide the event context in JSON format as custom parameters for Lambda to use. The event context provides information about the request being made in the event passed from S3 Object Lambda to Lambda. For more information about all the fields in the event context, see [Event context format and usage](olap-event-context.md). **To create an S3 Object Lambda Access Point** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose **Create Object Lambda Access Point**. 1. For **Object Lambda Access Point name**, enter the name that you want to use for the Object Lambda Access Point (for example, **tutorial-pii-object-lambda-accesspoint**). 1. For **Supporting Access Point**, enter or browse to the standard access point that you created in [Step 3](#ol-pii-step3) (for example, **tutorial-pii-access-point**), and then choose **Choose supporting Access Point**. 1. For **S3 APIs**, to retrieve objects from the S3 bucket for Lambda function to process, select **GetObject**. 1. For **Invoke Lambda function**, you can choose either of the following two options for this tutorial. + Choose **Choose from functions in your account** and choose the Lambda function that you deployed in [Step 4](#ol-pii-step4) (for example, **serverlessrepo-ComprehendPiiRedactionS3ObjectLambda**) from the **Lambda function** dropdown list. + Choose **Enter ARN**, and then enter the Amazon Resource Name (ARN) of the Lambda function that you created in [Step 4](#ol-pii-step4). 1. For **Lambda function version**, choose **\$1LATEST** (the latest version of the Lambda function that you deployed in [Step 4](#ol-pii-step4)). 1. (Optional) If you need your Lambda function to recognize and process GET requests with range and part number headers, select **Lambda function supports requests using range** and **Lambda function supports requests using part numbers**. Otherwise, clear these two check boxes. For more information about how to use range or part numbers with S3 Object Lambda, see [Working with Range and partNumber headers](range-get-olap.md). 1. (Optional) Under **Payload - *optional***, add JSON text to provide your Lambda function with additional information. A payload is optional JSON text that you can provide to your Lambda function as input for all invocations coming from a specific S3 Object Lambda Access Point. To customize the behaviors for multiple Object Lambda Access Points that invoke the same Lambda function, you can configure payloads with different parameters, thereby extending the flexibility of your Lambda function. For more information about payload, see [Event context format and usage](olap-event-context.md). 1. (Optional) For **Request metrics - *optional***, choose **Disable** or **Enable** to add Amazon S3 monitoring to your Object Lambda Access Point. Request metrics are billed at the standard Amazon CloudWatch rate. For more information, see [CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). 1. Under **Object Lambda Access Point policy - *optional***, keep the default setting. (Optional) You can set a resource policy. This resource policy grants the `GetObject` API permission to use the specified Object Lambda Access Point. 1. Keep the remaining settings set to the defaults, and choose **Create Object Lambda Access Point**. ## Step 6: Use the S3 Object Lambda Access Point to retrieve the redacted file Now, S3 Object Lambda is ready to redact PII data from your original file. **To use the S3 Object Lambda Access Point to retrieve the redacted file** When you request to retrieve a file through your S3 Object Lambda Access Point, you make a `GetObject` API call to S3 Object Lambda. S3 Object Lambda invokes the Lambda function to redact your PII data and returns the transformed data as the response to the standard S3 `GetObject` API call. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose the S3 Object Lambda Access Point that you created in [Step 5](#ol-pii-step5) (for example, **tutorial-pii-object-lambda-accesspoint**). 1. On the **Objects** tab of your S3 Object Lambda Access Point, select the file that has the same name (for example, `tutorial.txt`) as the one that you uploaded to the S3 bucket in [Step 2](#ol-pii-step2). This file should contain all the transformed data. 1. To view the transformed data, choose **Open** or **Download**. You should be able to see the redacted file, as shown in the following example. ``` Hello *********. Your AnyCompany Financial Services, LLC credit card account ******************* has a minimum payment of $24.53 that is due by *********. Based on your autopay settings, we will withdraw your payment on the due date from your bank account ********** with the routing number *********. Your latest statement was mailed to **********************************. After your payment is received, you will receive a confirmation text message at ************. If you have questions about your bill, AnyCompany Customer Service is available by phone at ************ or email at **********************. ``` ## Step 7: Clean up If you redacted your data through S3 Object Lambda only as a learning exercise, delete the AWS resources that you allocated so that you no longer accrue charges. **Topics** + [ ### Delete the Object Lambda Access Point ](#ol-pii-step8-delete-olap) + [ ### Delete the S3 access point ](#ol-pii-step8-delete-ap) + [ ### Delete the Lambda function ](#ol-pii-step8-delete-lambda-function) + [ ### Delete the CloudWatch log group ](#ol-pii-step8-delete-cloudwatch) + [ ### Delete the original file in the S3 source bucket ](#ol-pii-step8-delete-file) + [ ### Delete the S3 source bucket ](#ol-pii-step8-delete-bucket) + [ ### Delete the IAM role for your Lambda function ](#ol-pii-step8-delete-lambda-role) + [ ### Delete the customer managed policy for your IAM user ](#ol-pii-step8-delete-function-policy) + [ ### Delete the IAM user ](#ol-pii-step8-delete-user) ### Delete the Object Lambda Access Point 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Object Lambda Access Points**. 1. On the **Object Lambda Access Points** page, choose the option button to the left of the S3 Object Lambda Access Point that you created in [Step 5](#ol-pii-step5) (for example, **tutorial-pii-object-lambda-accesspoint**). 1. Choose **Delete**. 1. Confirm that you want to delete your Object Lambda Access Point by entering its name in the text field that appears, and then choose **Delete**. ### Delete the S3 access point 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Access Points**. 1. Navigate to the access point that you created in [Step 3](#ol-pii-step3) (for example, **tutorial-pii-access-point**), and choose the option button next to the name of the access point. 1. Choose **Delete**. 1. Confirm that you want to delete your access point by entering its name in the text field that appears, and then choose **Delete**. ### Delete the Lambda function 1. In the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/), choose **Functions** in the left navigation pane. 1. Choose the function that you created in [Step 4](#ol-pii-step4) (for example, **serverlessrepo-ComprehendPiiRedactionS3ObjectLambda**). 1. Choose **Actions**, and then choose **Delete**. 1. In the **Delete function** dialog box, choose **Delete**. ### Delete the CloudWatch log group 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the left navigation pane, choose **Log groups**. 1. Find the log group whose name ends with the Lambda function that you created in [Step 4](#ol-pii-step4) (for example, **serverlessrepo-ComprehendPiiRedactionS3ObjectLambda**). 1. Choose **Actions**, and then choose **Delete log group(s)**. 1. In the **Delete log group(s)** dialog box, choose **Delete**. ### Delete the original file in the S3 source bucket 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Bucket name** list, choose the name of the bucket that you uploaded the original file to in [Step 2](#ol-pii-step2) (for example, **tutorial-bucket**). 1. Select the check box to the left of the name of the object that you want to delete (for example, `tutorial.txt`). 1. Choose **Delete**. 1. On the **Delete objects** page, in the **Permanently delete objects?** section, confirm that you want to delete this object by entering **permanently delete** in the text box. 1. Choose **Delete objects**. ### Delete the S3 source bucket 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the option button next to the name of the bucket that you created in [Step 1](#ol-pii-step1) (for example, **tutorial-bucket**). 1. Choose **Delete**. 1. On the **Delete bucket** page, confirm that you want to delete the bucket by entering the bucket name in the text field, and then choose **Delete bucket**. ### Delete the IAM role for your Lambda function 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**, and then select the check box next to the role name that you want to delete. The role name starts with the name of the Lambda function that you deployed in [Step 4](#ol-pii-step4) (for example, **serverlessrepo-ComprehendPiiRedactionS3ObjectLambda**). 1. Choose **Delete**. 1. In the **Delete** dialog box, enter the role name in the text input field to confirm deletion. Then, choose **Delete**. ### Delete the customer managed policy for your IAM user 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policies**. 1. On the **Policies** page, enter the name of the customer managed policy that you created in the [Prerequisites](#ol-pii-prerequisites) (for example, **tutorial-serverless-application-repository**) in the search box to filter the list of policies. Select the option button next to the name of the policy that you want to delete. 1. Choose **Actions**, and then choose **Delete**. 1. Confirm that you want to delete this policy by entering its name in the text field that appears, and then choose **Delete**. ### Delete the IAM user 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Users**, and then select the check box next to the user name that you want to delete. 1. At the top of the page, choose **Delete**. 1. In the **Delete *user name*?** dialog box, enter the user name in the text input field to confirm the deletion of the user. Choose **Delete**. ## Next steps After completing this tutorial, you can further explore the following related use cases: + You can create multiple S3 Object Lambda Access Points and enable them with prebuilt Lambda functions that are configured differently to redact specific types of PII depending on the data accessors' business needs. Each type of user assumes an IAM role and only has access to one S3 Object Lambda Access Point (managed through IAM policies). Then, you attach each `ComprehendPiiRedactionS3ObjectLambda` Lambda function configured for a different redaction use case to a different S3 Object Lambda Access Point. For each S3 Object Lambda Access Point, you can have a supporting S3 access point to read data from an S3 bucket that stores the shared dataset. For more information about how to create an S3 bucket policy that allows users to read from the bucket only through S3 access points, see [Configuring IAM policies for using access points](access-points-policies.md). For more information about how to grant a user permission to access the Lambda function, the S3 access point, and the S3 Object Lambda Access Point, see [Configuring IAM policies for Object Lambda Access Points](olap-policies.md). + You can build your own Lambda function and use S3 Object Lambda with your customized Lambda function to meet your specific data needs. For example, to explore various data values, you can use S3 Object Lambda and your own Lambda function that uses additional [Amazon Comprehend features](https://aws.amazon.com/comprehend/features/), such as entity recognition, key phrase recognition, sentiment analysis, and document classification, to process data. You can also use S3 Object Lambda together with [Amazon Comprehend Medical](https://aws.amazon.com/comprehend/medical/), a HIPAA-eligible NLP service, to analyze and extract data in a context-aware manner. For more information about how to transform data with S3 Object Lambda and your own Lambda function, see [Tutorial: Transforming data for your application with S3 Object Lambda](tutorial-s3-object-lambda-uppercase.md). # Debugging and troubleshooting S3 Object Lambda **Note** As of November 7th, 2025, S3 Object Lambda is available only to existing customers that are currently using the service as well as to select AWS Partner Network (APN) partners. For capabilities similar to S3 Object Lambda, learn more here - [Amazon S3 Object Lambda availability change](https://docs.aws.amazon.com/AmazonS3/latest/userguide/amazons3-ol-change.html). Requests to Amazon S3 Object Lambda access points might result in a new error response when something goes wrong with the Lambda function invocation or execution. These errors follow the same format as standard Amazon S3 errors. For information about S3 Object Lambda errors, see [S3 Object Lambda Error Code List](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#S3ObjectLambdaErrorCodeList) in the *Amazon Simple Storage Service API Reference*. For more information about general Lambda function debugging, see [Monitoring and troubleshooting Lambda applications](https://docs.aws.amazon.com/lambda/latest/dg/lambda-monitoring.html ) in the *AWS Lambda Developer Guide*. For information about standard Amazon S3 errors, see [Error Responses](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html) in the *Amazon Simple Storage Service API Reference*. You can enable request metrics in Amazon CloudWatch for your Object Lambda Access Points. These metrics help you monitor the operational performance of your access point. You can enable request metrics during or after creation of your Object Lambda Access Point. For more information, see [S3 Object Lambda request metrics in CloudWatch](metrics-dimensions.md#olap-cloudwatch-metrics). To get more granular logging about requests made to your Object Lambda Access Points, you can enable AWS CloudTrail data events. For more information, see [Logging data events for trails](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html) in the *AWS CloudTrail User Guide*. For S3 Object Lambda tutorials, see the following: + [Tutorial: Transforming data for your application with S3 Object Lambda](tutorial-s3-object-lambda-uppercase.md) + [Tutorial: Detecting and redacting PII data with S3 Object Lambda and Amazon Comprehend](tutorial-s3-object-lambda-redact-pii.md) + [Tutorial: Using S3 Object Lambda to dynamically watermark images as they are retrieved](https://aws.amazon.com/getting-started/hands-on/amazon-s3-object-lambda-to-dynamically-watermark-images/?ref=docs_gateway/amazons3/transforming-objects.html) For more information about standard access points, see [Managing access to shared datasets with access points](access-points.md). For information about working with buckets, see [General purpose buckets overview](UsingBucket.md). For information about working with objects, see [Amazon S3 objects overview](UsingObjects.md). # Performing object operations in bulk with Batch Operations You can use S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. S3 Batch Operations can perform a single operation on lists of Amazon S3 objects that you specify. A single job can perform a specified operation on billions of objects containing exabytes of data. Amazon S3 tracks progress, sends notifications, and stores a detailed completion report of all actions, providing a fully managed, auditable, and serverless experience. You can use S3 Batch Operations through the Amazon S3 console, AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. Use S3 Batch Operations to copy objects, update server-side encryption of objects, and set object tags or access control lists (ACLs). You can also initiate object restores from S3 Glacier Flexible Retrieval or invoke an AWS Lambda function to perform custom actions using your objects. You can perform these operations on a custom list of objects, or you can use an Amazon S3 Inventory report to easily generate lists of objects. Amazon S3 Batch Operations use the same Amazon S3 API operations that you already use with Amazon S3. **Note** For more information about using the Amazon S3 Express One Zone storage class with directory buckets, see [S3 Express One Zone](directory-bucket-high-performance.md#s3-express-one-zone) and [Working with directory buckets](directory-buckets-overview.md). For more information about using Batch Operations with S3 Express One Zone and directory buckets, see [Using Batch Operations with directory buckets](directory-buckets-objects-Batch-Ops.md). ## S3 Batch Operations basics You can use S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. S3 Batch Operations can run a single operation or action on lists of Amazon S3 objects that you specify. ### Terminology This section uses the terms *manifests*, *jobs*, *operations*, and *tasks*, which are defined as follows: **Manifest** A manifest is an Amazon S3 object that contains the object keys that you want Amazon S3 to act upon. If you want to create a Batch Operations job, you must supply a manifest. Your user-generated manifest must contain the bucket name, object key, and optionally, the object version for each object. If you supply a user-generated manifest, it must be in the form of an Amazon S3 Inventory report or a CSV file. You can also have Amazon S3 generate a manifest automatically based on object filter criteria that you specify when you create your job. This option is available for Batch Operations jobs that you create by using the Amazon S3 console, or for any job type that you create by using the AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. **Job** A job is the basic unit of work for S3 Batch Operations. A job contains all of the information necessary to run the specified operation on the objects listed in the manifest. After you provide this information and request that the job begin, the job performs the operation for each object in the manifest. **Operation** The operation is the type of API [action](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html), such as copying objects, that you want the Batch Operations job to run. Each job performs a single type of operation across all objects that are specified in the manifest. **Task** A task is the unit of execution for a job. A task represents a single call to an Amazon S3 or AWS Lambda API operation to perform the job's operation on a single object. Over the course of a job's lifetime, S3 Batch Operations create one task for each object specified in the manifest. ### How an S3 Batch Operations job works A job is the basic unit of work for S3 Batch Operations. A job contains all of the information necessary to run the specified operation on a list of objects. To create a job, you give S3 Batch Operations a list of objects and specify the action to perform on those objects. For information about the operations that S3 Batch Operations supports, see [Operations supported by S3 Batch Operations](batch-ops-operations.md). A batch job performs a specified operation on every object that's included in its *manifest*. A manifest lists the objects that you want a batch job to process and it is stored as an object in a bucket. You can use a comma-separated values (CSV)-formatted [Cataloging and analyzing your data with S3 Inventory](storage-inventory.md) report as a manifest, which makes it easy to create large lists of objects located in a bucket. You can also specify a manifest in a simple CSV format that enables you to perform batch operations on a customized list of objects contained within a single bucket. After you create a job, Amazon S3 processes the list of objects in the manifest and runs the specified operation against each object. While a job is running, you can monitor its progress programmatically or through the Amazon S3 console. You can also configure a job to generate a completion report when it finishes. The completion report describes the results of each task that was performed by the job. For more information about monitoring jobs, see [Managing S3 Batch Operations jobs](batch-ops-managing-jobs.md). There are costs associated with S3 Batch Operations. You are billed for creating Batch Operations jobs, including jobs that are canceled before completion. For more information, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/). S3 Batch Operations jobs by default can process up to 4 billion objects for all operations. Specifically Copy, Object Tagging, Object Lock, invoking an AWS Lambda function, and Batch Replication jobs can support up to 20 billion objects. There is a limit of 6 active Batch Replication jobs per AWS account. To get started creating a Batch Operations job, see [Creating an S3 Batch Operations job](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html). ## S3 Batch Operations tutorial The following tutorial presents complete end-to-end procedures for some Batch Operations tasks. + [Tutorial: Batch-transcoding videos with S3 Batch Operations](tutorial-s3-batchops-lambda-mediaconvert-video.md) # Granting permissions for Batch Operations Before creating and running S3 Batch Operations jobs, you must grant required permissions. To create an Amazon S3 Batch Operations job, the `s3:CreateJob` user permission is required. The same entity that creates the job must also have the `iam:PassRole` permission to pass the AWS Identity and Access Management (IAM) role that's specified for the job to Batch Operations. The following sections provide information about creating an IAM role and attaching policies. For general information about specifying IAM resources, see [IAM JSON policy elements: Resource](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_resource.html) in the *IAM User Guide*. **Topics** + [ ## Creating an S3 Batch Operations IAM role ](#batch-ops-iam-role-policies-create) + [ ## Attaching permissions policies ](#batch-ops-iam-role-policies-perm) ## Creating an S3 Batch Operations IAM role Amazon S3 must have permissions to perform S3 Batch Operations on your behalf. You grant these permissions through an AWS Identity and Access Management (IAM) role. When you create an S3 Batch Operations job, you specify the IAM role that you want the job to use. This can be an existing IAM role. Or, if you use the Amazon S3 console to create the job, it can be an IAM role that Amazon S3 creates for you. If you choose to let Amazon S3 create the IAM role for you, it automatically creates and attaches trust and permissions policies to the role. The trust policy allows the S3 Batch Operations service principal (`batchoperations.s3.amazonaws.com`) to assume the role. The permissions policy allows all the requisite actions for running the job, based on the settings that you specify for the job. For example, if you configure a job to copy objects from one bucket to another bucket in your AWS account, the permissions policy allows actions such as `s3:GetObject` and `s3:PutObject`. You can review the trust and permissions policies for the role before you submit the job. This option is available only if you use the Amazon S3 console to create a job, and you configure the job to use an S3 generated object list that uses filters or is based on a replication configuration. After you submit the job, the IAM role persists in your account. You can then use it again for subsequent jobs that perform the same operation, or delete it when the job finishes running. If you prefer to create the IAM role manually, the policy examples in this section can help you create the role. For more information about creating and configuring roles, see [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) in the *IAM User Guide*. For information about permissions for S3 API operations by S3 resource type, see [Required permissions for Amazon S3 API operations](using-with-s3-policy-actions.md). For additional examples, see [Controlling permissions for Batch Operations using job tags](batch-ops-job-tags-examples.md) and [Copying objects using S3 Batch Operations](batch-ops-examples-copy.md). In your IAM policies, you can also use condition keys to filter access permissions for S3 Batch Operations jobs. For more information and a complete list of Amazon S3 specific condition keys, see [ Actions, resources, and condition keys for Amazon S3](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html) in the *Service Authorization Reference*. For more information about the permissions to S3 API operations by S3 resource types, see [Required permissions for Amazon S3 API operations](using-with-s3-policy-actions.md). The following video includes how to set up IAM permissions for Batch Operations jobs using the Amazon S3 console. [![AWS Videos](http://img.youtube.com/vi/https://www.youtube.com/embed/qpwHUrwAiUI//0.jpg)](http://www.youtube.com/watch?v=https://www.youtube.com/embed/qpwHUrwAiUI/) ### Trust policy To allow the S3 Batch Operations service principal to assume the IAM role, attach the following trust policy to the role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Principal":{ "Service":"batchoperations.s3.amazonaws.com" }, "Action":"sts:AssumeRole" } ] } ``` ------ ## Attaching permissions policies Depending on the type of operations, you can attach one of the following policies. Before you configure permissions, note the following: + Regardless of the operation, Amazon S3 needs permissions to read your manifest object from your S3 bucket and optionally write a report to your bucket. Therefore, all of the following policies include these permissions. + For Amazon S3 Inventory report manifests, S3 Batch Operations requires permission to read the manifest.json object and all associated CSV data files. + Version-specific permissions such as `s3:GetObjectVersion` are only required when you are specifying the version ID of the objects. + If you are running S3 Batch Operations on encrypted objects, the IAM role must also have access to the AWS KMS keys used to encrypt them. + If you submit an inventory report manifest that's encrypted with AWS KMS, your IAM policy must include the permissions `"kms:Decrypt"` and `"kms:GenerateDataKey"` for the manifest.json object and all associated CSV data files. + If the Batch Operations job generates a manifest in a bucket that has access control lists (ACLs) enabled and is in a different AWS account, you must grant the `s3:PutObjectAcl` permission in the IAM policy of the IAM role configured for the batch job. If you don't include this permission, the batch job fails with the error `Error occurred when preparing manifest: Failed to write manifest`. ### Copy objects: PutObject ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:PutObjectTagging" ], "Effect": "Allow", "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" }, { "Action": [ "s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectTagging", "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::amzn-s3-demo-source-bucket", "arn:aws:s3:::amzn-s3-demo-source-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Replace object tagging: PutObjectTagging ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "s3:PutObjectTagging", "s3:PutObjectVersionTagging" ], "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect":"Allow", "Action":[ "s3:PutObject" ], "Resource":[ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Delete object tagging: DeleteObjectTagging ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:DeleteObjectTagging", "s3:DeleteObjectVersionTagging" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Replace access control list: PutObjectAcl ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObjectAcl", "s3:PutObjectVersionAcl" ], "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Restore objects: RestoreObject ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "s3:RestoreObject" ], "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect":"Allow", "Action":[ "s3:PutObject" ], "Resource":[ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Apply Object Lock retention: PutObjectRetention ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:GetBucketObjectLockConfiguration", "Resource": [ "arn:aws:s3:::amzn-s3-demo-destination-bucket" ] }, { "Effect": "Allow", "Action": [ "s3:PutObjectRetention", "s3:BypassGovernanceRetention" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Apply Object Lock legal hold: PutObjectLegalHold ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:GetBucketObjectLockConfiguration", "Resource": [ "arn:aws:s3:::amzn-s3-demo-destination-bucket" ] }, { "Effect": "Allow", "Action": "s3:PutObjectLegalHold", "Resource": [ "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Replicate existing objects: InitiateReplication with an S3 generated manifest Use this policy if you're using and storing an S3 generated manifest. For more information about using Batch Operations to replicate existing objects, see [Replicating existing objects with Batch Replication](s3-batch-replication-batch.md). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Action":[ "s3:InitiateReplication" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::amzn-s3-demo-source-bucket/*" ] }, { "Action":[ "s3:GetReplicationConfiguration", "s3:PutInventoryConfiguration" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::amzn-s3-demo-source-bucket" ] }, { "Action":[ "s3:GetObject", "s3:GetObjectVersion" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect":"Allow", "Action":[ "s3:PutObject" ], "Resource":[ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*", "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] } ] } ``` ------ ### Replicate existing objects: InitiateReplication with a user manifest Use this policy if you're using a user supplied manifest. For more information about using Batch Operations to replicate existing objects, see [Replicating existing objects with Batch Replication](s3-batch-replication-batch.md). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Action":[ "s3:InitiateReplication" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::amzn-s3-demo-source-bucket/*" ] }, { "Action":[ "s3:GetObject", "s3:GetObjectVersion" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect":"Allow", "Action":[ "s3:PutObject" ], "Resource":[ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] } ``` ------ ### Compute checksum: Allow `GetObject`, `GetObjectVersion`, `RestoreObject`, and `PutObject` Use this policy if you're trying to use the **Compute checksum** operation with S3 Batch Operations. Permissions for `GetObject`, `GetObjectVersion`, and `RestoreObject` are required to obtain and read the bytes of stored data. Replace the user input placeholders with your own information. For more information about **Compute checksum**, see [Checking object integrity for data at rest in Amazon S3](checking-object-integrity-at-rest.md). ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:RestoreObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket1/*" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket2/*" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket3/*" ] } ] } ``` ### Update object encryption You must attach the following permissions policy to allow Batch Operations to read a manifest, update the encryption type of your objects, and write a completion report. To use this permissions policy, replace the *`user input placeholders`* with your own information. For more information about using this operation and the permissions that you must attach to the role used by your IAM principal, see [Update object encryption](batch-ops-update-encryption.md). ``` { "Version": "2012-10-17", "Statement": [{ "Sid": "S3BatchOperationsUpdateEncryption", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:PutObject", "s3:UpdateObjectEncryption" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-target" "arn:aws:s3:::amzn-s3-demo-bucket-target/*" ] }, { "Sid": "S3BatchOperationsPolicyForManifestFile", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-manifest/*" ] }, { "Sid": "S3BatchOperationsPolicyForCompletionReport", "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-completion-report/*" ] }, { "Sid": "S3BatchOperationsPolicyManifestGeneration", "Effect": "Allow", "Action": [ "s3:PutInventoryConfiguration" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-target" ] } { "Sid": "AllowKMSOperationsForS3BatchOperations", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey", "kms:Encrypt", "kms:ReEncrypt*" ], "Resource": [ "arn:aws:kms:us-east-1:111122223333:key/01234567-89ab-cdef-0123-456789abcdef" ] } ] } ``` # Creating an S3 Batch Operations job With Amazon S3 Batch Operations, you can perform large-scale batch operations on a list of specific Amazon S3 objects. This section describes the information that you need to create an S3 Batch Operations job and the results of a `CreateJob` request. It also provides instructions for creating a Batch Operations job by using the Amazon S3 console, AWS Command Line Interface (AWS CLI), and AWS SDK for Java. When you create an S3 Batch Operations job, you can request a completion report for all tasks or only failed tasks. As long as at least one task has been invoked successfully, S3 Batch Operations generates a report for jobs that have been completed, have failed, or have been canceled. For more information, see [Examples: S3 Batch Operations completion reports](batch-ops-examples-reports.md). The following video provides a brief demonstration of how to create a Batch Operations job by using the Amazon S3 console. [![AWS Videos](http://img.youtube.com/vi/https://www.youtube.com/embed/qpwHUrwAiUI//0.jpg)](http://www.youtube.com/watch?v=https://www.youtube.com/embed/qpwHUrwAiUI/) **Topics** + [ ## Batch Operations job request elements ](#batch-ops-create-job-request-elements) + [ ## Specifying a manifest ](#specify-batchjob-manifest) + [ ## Generating an object list automatically and saving it as a manifest file ](#automatically-generate-manifest-file) + [ ## Creating a manifest file ](#create-manifest-file) + [ ## Using an existing manifest ](#specify-existing-manifest-file) + [ ## Creating a job ](#to-create-batch-ops-job) + [ ## Job responses ](#batch-ops-create-job-response-elements) ## Batch Operations job request elements To create an S3 Batch Operations job, you must provide the following information: **Operation** Specify the operation that you want S3 Batch Operations to run against the objects in the manifest. Each operation type accepts parameters that are specific to that operation. With Batch Operations, you can perform an operation in bulk, with the same results as if you performed that operation one-by-one on each object. **Manifest** A *manifest* is an Amazon S3 object list that contains the object keys that you want Amazon S3 to act upon. You can use the following methods to specify a manifest for a Batch Operations job: + Direct Batch Operations to generate an object list based on metadata that you specify. You can save this list as a manifest file and use it when you create your job. This option is available for any job type that you create by using the Amazon S3 console, the AWS CLI, AWS SDKs, or Amazon S3 REST API. + Generate an object list automatically based on an existing replication configuration. You can save this list as a manifest file and use it again for future jobs. + Create a new manifest file manually. + Use an existing manifest. + Regardless of how you specify the objects to work on, the manifest itself must be stored in a general purpose bucket. Batch Operations can't import existing manifests from, or save generated object lists as manifests to directory buckets. Objects described within the manifest, however, can be stored in directory buckets. For more information, see [Directory buckets](https://docs.aws.amazon.com//AmazonS3/latest/userguide/directory-buckets-overview.html). + If the objects in your manifest are in a versioned bucket, specifying the version IDs for the objects directs Batch Operations to perform the operation on a specific version. If no version IDs are specified, Batch Operations performs the operation on the latest version of the objects. If your manifest includes a version ID field, you must provide a version ID for all objects in the manifest. For more information, see [Specifying a manifest](#specify-batchjob-manifest). **Priority** Use job priorities to indicate the relative priority of this job to others running in your account. A higher number indicates higher priority. Job priorities only have meaning relative to the priorities that are set for other jobs in the same account and Region. You can choose whatever numbering system works for you. For example, you might want to assign all **Restore** (`RestoreObject`) jobs a priority of 1, all **Copy** (`CopyObject`) jobs a priority of 2, and all **Replace access control lists (ACLs)** (`PutObjectAcl`) jobs a priority of 3. S3 Batch Operations prioritizes jobs according to priority numbers, but strict ordering isn't guaranteed. Therefore, don't use job priorities to ensure that any one job starts or finishes before any other job. If you must ensure strict ordering, wait until one job has finished before starting the next. **RoleArn** Specify an AWS Identity and Access Management (IAM) role to run the job. The IAM role that you use must have sufficient permissions to perform the operation specified in the job. For example, to run a `CopyObject` job, the IAM role must have the `s3:GetObject` permission for the source bucket and the `s3:PutObject` permission for the destination bucket. The role also needs permissions to read the manifest and write the completion report. The IAM role can be an existing role. Or, if you use the Amazon S3 console to create the job, it can be an IAM role that Amazon S3 creates automatically for you. For more information, see [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md). For more information about IAM roles, see [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) in the *IAM User Guide*. For more information about Amazon S3 permissions, see [Policy actions for Amazon S3](security_iam_service-with-iam.md#security_iam_service-with-iam-id-based-policies-actions). Batch Operations jobs that perform actions on directory buckets require specific permissions. For more information, see [AWS Identity and Access Management (IAM) for S3 Express One Zone](https://docs.aws.amazon.com//AmazonS3/latest/userguide/s3-express-security-iam.html). **Report** Specify whether you want S3 Batch Operations to generate a completion report. If you request a completion report, you must also provide the parameters for the report in this element. The following information is required: + The bucket where you want to store the report **Note** The report must be stored in a general purpose bucket. Batch Operations can't save reports to directory buckets. For more information, see [Directory buckets](https://docs.aws.amazon.com//AmazonS3/latest/userguide/directory-buckets-overview.html). + The format of the report + Whether you want the report to include the details of all tasks or only failed tasks + An optional prefix string If the `CreateJob.Report.ExpectedBucketOwner` field is supplied, it requires that the completion report bucket owner matches. If it doesn't match, then the job fails. Completion reports are always encrypted with server-side encryption with Amazon S3 managed keys (SSE-S3). **Tags (optional)** You can label and control access to your S3 Batch Operations jobs by adding *tags*. You can use tags to identify who is responsible for a Batch Operations job, or to control how users interact with Batch Operations jobs. The presence of job tags can grant or limit a user's ability to cancel a job, activate a job in the confirmation state, or change a job's priority level. For example, you could grant a user permission to invoke the `CreateJob` operation, provided that the job is created with the tag `"Department=Finance"`. You can create jobs with tags attached to them, and you can add tags to jobs after you create them. For more information, see [Controlling access and labeling jobs using tags](batch-ops-job-tags.md). **Description (optional)** To track and monitor your job, you can also provide a description of up to 256 characters. Amazon S3 includes this description whenever it returns information about a job or displays job details on the Amazon S3 console. You can then easily sort and filter jobs according to the descriptions that you assigned. Descriptions don't need to be unique, so you can use descriptions as categories (for example, "Weekly Log Copy Jobs") to help you track groups of similar jobs. ## Specifying a manifest A *manifest* is an Amazon S3 object list that contains the object keys that you want Amazon S3 to act upon. You can use the following methods to specify a manifest for a Batch Operations job: + Direct Batch Operations to generate an object list based on metadata that you specify. You can save this list as a manifest and use it when you create your job. This option is available for any job type that you create by using the Amazon S3 console, the AWS CLI, AWS SDKs, or Amazon S3 REST API. + Generate an object list automatically based on an existing replication configuration. You can save this list as a manifest and use it again for future jobs. + Create a new manifest file manually. + Use an existing manifest. **Note** Amazon S3 Batch Operations does not support cross-Region object list generation. Regardless of how you specify the objects to work on, the manifest itself must be stored in a general purpose bucket. Batch Operations can't import existing manifests from, or save generated object lists as manifests to directory buckets. Objects described within the manifest, however, can be stored in directory buckets. For more information, see [Directory buckets](https://docs.aws.amazon.com//AmazonS3/latest/userguide/directory-buckets-overview.html). ## Generating an object list automatically and saving it as a manifest file You can direct Amazon S3 to generate an object list automatically based on metadata that you specify. You can save this list as a manifest and use it when you create your job. This option is available for any job type that you create by using the Amazon S3 console, the AWS CLI, AWS SDKs, or Amazon S3 REST API. To generate an object list automatically and save it as a manifest file, you specify the following elements as part of your job creation request: + Information about the bucket that contains your source objects, including the bucket owner and Amazon Resource Name (ARN). + Information about the manifest output, including a flag to create a manifest file, the output bucket owner, the ARN, the prefix, the file format, and the encryption type. + Optional criteria to filter objects by their creation date, key name, size, encryption type, KMS key ARN, Bucket Key and storage class, In the case of replication jobs, you can also use tags to filter objects. ### Object filter criteria To filter the list of objects to be included in an automatically generated object list, you can specify the following criteria. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestGeneratorFilter.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestGeneratorFilter.html) in the *Amazon S3 API Reference*. **CreatedAfter** If provided, the generated manifest includes only source bucket objects that were created after this time. **CreatedBefore** If provided, the generated manifest includes only source bucket objects that were created before this time. **EligibleForReplication** If provided, the generated manifest includes objects only if they are eligible for replication according to the replication configuration on the source bucket. **KeyNameConstraint** If provided, the generated manifest includes only source bucket objects whose object keys match the string constraints specified for **MatchAnySubstring**, **MatchAnyPrefix**, and **MatchAnySuffix**. **MatchAnySubstring** – If provided, the generated manifest includes objects if the specified string appears anywhere within the object key string. **MatchAnyPrefix** – If provided, the generated manifest includes objects if the specified string appears at the start of the object key string. **MatchAnySuffix** – If provided, the generated manifest includes objects if the specified string appears at the end of the object key string. **MatchAnyObjectEncryption** If provided, the generated object list saved as a manifest file includes only source bucket objects with the indicated server-side encryption type (SSE-S3, SSE-KMS, DSSE-KMS, SSE-C, or NOT-SSE). If you select SSE- KMS or DSSE-KMS, you can optionally further filter your results by specifying a specific KMS key ARN. If you select SSE-KMS, you can also optionally further filter your results by Bucket Key enabled status. To improve manifest generation performance when using the `KmsKeyArn` filter, use the filter with other object metadata filters, such as `MatchAnyPrefix`, `CreatedAfter`, or `MatchAnyStorageClass`. **MatchAnyStorageClass** If provided, the generated manifest includes only source bucket objects that are stored with the specified storage class. **ObjectReplicationStatuses** If provided, the generated manifest includes only source bucket objects that have one of the specified replication statuses. **ObjectSizeGreaterThanBytes** If provided, the generated manifest includes only source bucket objects whose file size is greater than the specified number of bytes. **ObjectSizeLessThanBytes** If provided, the generated manifest includes only source bucket objects whose file size is less than the specified number of bytes. **Note** You can't clone most jobs that have automatically generated object lists saved as manifests. Batch replication jobs can be cloned, except when they use the `KeyNameConstraint`, `MatchAnyStorageClass`, `ObjectSizeGreaterThanBytes`, or `ObjectSizeLessThanBytes` manifest filter criteria. The syntax for specifying manifest criteria varies depending on the method that you use to create your job. For examples, see [Creating a job](#to-create-batch-ops-job). ## Creating a manifest file To create a manifest file manually, you specify the manifest object key, ETag (entity tag), and optional version ID in a CSV-formatted list. The contents of the manifest must be URL-encoded. By default, Amazon S3 automatically uses server-side encryption with Amazon S3 managed keys (SSE-S3) to encrypt a manifest that's uploaded to an Amazon S3 bucket. Manifests that use server-side encryption with customer-provided keys (SSE-C) are not supported. Manifests that use server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS) are supported only when you're using CSV-formatted inventory reports. Using a manually created manifest with AWS KMS is not supported. Your manifest must contain the bucket name, object key, and optionally, the object version for each object. Any other fields in the manifest are not used by S3 Batch Operations. **Note** If the objects in your manifest are in a versioned bucket, specifying the version IDs for the objects directs Batch Operations to perform the operation on a specific version. If no version IDs are specified, Batch Operations performs the operation on the latest version of the objects. If your manifest includes a version ID field, you must provide a version ID for all objects in the manifest. The following is an example manifest in CSV format without version IDs. ``` amzn-s3-demo-bucket1,objectkey1 amzn-s3-demo-bucket1,objectkey2 amzn-s3-demo-bucket1,objectkey3 amzn-s3-demo-bucket1,photos/jpgs/objectkey4 amzn-s3-demo-bucket1,photos/jpgs/newjersey/objectkey5 amzn-s3-demo-bucket1,object%20key%20with%20spaces ``` The following is an example manifest in CSV format that includes version IDs. ``` amzn-s3-demo-bucket1,objectkey1,PZ9ibn9D5lP6p298B7S9_ceqx1n5EJ0p amzn-s3-demo-bucket1,objectkey2,YY_ouuAJByNW1LRBfFMfxMge7XQWxMBF amzn-s3-demo-bucket1,objectkey3,jbo9_jhdPEyB4RrmOxWS0kU0EoNrU_oI amzn-s3-demo-bucket1,photos/jpgs/objectkey4,6EqlikJJxLTsHsnbZbSRffn24_eh5Ny4 amzn-s3-demo-bucket1,photos/jpgs/newjersey/objectkey5,imHf3FAiRsvBW_EHB8GOu.NHunHO1gVs amzn-s3-demo-bucket1,object%20key%20with%20spaces,9HkPvDaZY5MVbMhn6TMn1YTb5ArQAo3w ``` ## Using an existing manifest You can specify an existing manifest to create a Batch Operations job by using one of the following two formats: + **Amazon S3 Inventory report** – Must be a CSV-formatted Amazon S3 Inventory report. You must specify the `manifest.json` file that is associated with the inventory report. For more information about inventory reports, see [Cataloging and analyzing your data with S3 Inventory](storage-inventory.md). If the inventory report includes version IDs, S3 Batch Operations operates on the specific object versions. **Note** S3 Batch Operations supports CSV *inventory reports *that are encrypted with SSE-KMS. If you submit an inventory report manifest that's encrypted with SSE-KMS, your IAM policy must include the permissions `"kms:Decrypt"` and `"kms:GenerateDataKey"` for the `manifest.json` object and all associated CSV data files. + **CSV file** – Each row in the file must include the bucket name, object key, and optionally, the object version. Object keys must be URL-encoded, as shown in the following examples. The manifest must either include version IDs for all objects or omit version IDs for all objects. For more information about the CSV manifest format, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestSpec.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestSpec.html) in the *Amazon Simple Storage Service API Reference*. **Note** S3 Batch Operations doesn't support CSV *manifest files* that are encrypted with SSE-KMS. **Important** When you're using a manually created manifest and a versioned bucket, we recommend that you specify the version IDs for the objects. When you create a job, S3 Batch Operations parses the entire manifest before running the job. However, it doesn't take a "snapshot" of the state of the bucket. Because manifests can contain billions of objects, jobs might take a long time to run, which can affect which version of an object that the job acts upon. Suppose that you overwrite an object with a new version while a job is running and you didn't specify a version ID for that object. In this case, Amazon S3 performs the operation on the latest version of the object, not on the version that existed when you created the job. The only way to avoid this behavior is to specify version IDs for the objects that are listed in the manifest. ## Creating a job You can create S3 Batch Operations jobs by using the Amazon S3 console, AWS CLI, AWS SDKs, or Amazon S3 REST API. For more information about creating a job request, see [Batch Operations job request elements](#batch-ops-create-job-request-elements). **Prerequisites** Before you create a Batch Operations job, confirm that you have configured the relevant permissions. For more information, see [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md). ### Using the S3 console **To create a Batch Operations job using the S3 console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. Choose **Batch Operations** on the left navigation pane of the Amazon S3 console. 1. Choose **Create job**. 1. Under **Choose Region and scope**, choose and view the AWS Region where you want to create your job. **Note** For copy operations, you must create the job in the same Region as the destination bucket. For all other operations, you must create the job in the same Region as the objects in the manifest. 1. For **Scope**, specify the list of objects that your Batch Operations job will act upon. Under **Object list**, you can choose to generate a manifest using an object list, generate a manifest using a replication configuration, or use an existing manifest. + If you choose **Generate an object list**, an object list is automatically generated based on the source location and metadata that you specify. You can save this list as a manifest and use it again for future jobs. **Note** To generate an object list, you must have the `s3:PutInventoryConfiguration` permission. The source bucket must be a general purpose bucket. + If you choose **Use an existing manifest**, you can import an object list from an existing manifest. A manifest is an S3 Inventory report or CV file that lists the specific objects that you want Batch Operations to act upon. + If you choose **Use a replication configuration**, you can generate an object list automatically based on an existing replication configuration. You can save this list as a manifest and use it again for future jobs. For this example, choose **Generate an object list**. 1. For **Source account**, choose the account that owns the source objects. 1. Under **Source**, enter the path to your source, for example, `s3://`*amzn-s3-demo-bucket*. 1. Under **Object filters**, you can use filters to filter by any portion of the object key or to filter by the end of the object key. The **Object key filters** assist in refining the list of objects to be used in the manifest. For **Object metadata filters**, choose filters to further define the scope of objects to include in the manifest. 1. Under **Choose operation**, choose the operation type that you want to perform on all objects listed in the manifest. If your manifest references objects stored in a directory bucket, only use the copy or invoke AWS Lambda function operations. All other operations aren't supported. 1. After selecting your operation type, choose **Next**. 1. Fill out the information for **Configure additional options**. For **Permissions**, specify the AWS Identity and Access Management (IAM) role that you want the job to use. This can be an existing role, or a role that Amazon S3 creates automatically for you. For more information, see [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md). Amazon S3 can create the role for you if you configured the job to use an S3 generated object list with filters or an object list based on a replication configuration. 1. When you finish configuring the additional options, choose **Next**. 1. For **Review**, verify the settings. If you need to make changes, choose **Previous**. Otherwise, you can choose **Submit job**. ### Using the AWS CLI To create your Batch Operations job with the AWS CLI, choose one of the following examples, depending on whether you're specifying an existing manifest or generating a manifest automatically. ------ #### [ Specify manifest ] The following example shows how to use the AWS CLI to create an S3 Batch Operations `S3PutObjectTagging` job that acts on objects that are listed in an existing manifest file. **To create a Batch Operations `S3PutObjectTagging` job by specifying a manifest** 1. Use the following commands to create an AWS Identity and Access Management (IAM) role, and then create an IAM policy to assign the relevant permissions. The following role and policy grant Amazon S3 permission to add object tags, which you will need when you create the job in a subsequent step. 1. Use the following example command to create an IAM role for Batch Operations to use. To use this example command, replace `S3BatchJobRole` with the name that you want to give to the role. ``` aws iam create-role \ --role-name S3BatchJobRole \ --assume-role-policy-document '{ "Version": "2012-10-17" , "Statement":[ { "Effect":"Allow", "Principal":{ "Service":"batchoperations.s3.amazonaws.com" }, "Action":"sts:AssumeRole" } ] }' ``` Record the role's Amazon Resource Name (ARN). You will need the ARN when you create a job. 1. Use the following example command to create an IAM policy with the necessary permissions and attach it to the IAM role that you created in the previous step. For more information about the necessary permissions, see [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md). **Note** Batch Operations jobs that perform actions on directory buckets require specific permissions. For more information, see [AWS Identity and Access Management (IAM) for S3 Express One Zone](https://docs.aws.amazon.com//AmazonS3/latest/userguide/s3-express-security-iam.html). To use this example command, replace the `user input placeholders` as follows: + Replace `S3BatchJobRole` with the name of your IAM role. Make sure that this name matches the name that you used earlier. + Replace `PutObjectTaggingBatchJobPolicy` with the name that you want to give your IAM policy. + Replace `amzn-s3-demo-destination-bucket` with the name of the bucket that contains the objects that you want to apply tags to. + Replace *`amzn-s3-demo-manifest-bucket`* with the name of the bucket that contains the manifest. + Replace *`amzn-s3-demo-completion-report-bucket`* with the name of the bucket where you want the completion report to be delivered to. ``` aws iam put-role-policy \ --role-name S3BatchJobRole \ --policy-name PutObjectTaggingBatchJobPolicy \ --policy-document '{ "Version": "2012-10-17" , TCX5-2025-waiver;, "Statement":[ { "Effect":"Allow", "Action":[ "s3:PutObjectTagging", "s3:PutObjectVersionTagging" ], "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket", "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect":"Allow", "Action":[ "s3:PutObject", "s3:GetBucketLocation" ], "Resource":[ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket", "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" ] } ] }' ``` 1. Use the following example command to create an `S3PutObjectTagging` job. The `manifest.csv` file provides a list of bucket and object key values. The job applies the specified tags to the objects that are identified in the manifest. The `ETag` is the ETag of the `manifest.csv` object, which you can get from the Amazon S3 console. This request specifies the `no-confirmation-required` parameter, so that you can run the job without having to confirm it with the `update-job-status` command. For more information, see [https://docs.aws.amazon.com/cli/latest/reference/s3control/create-job.html](https://docs.aws.amazon.com/cli/latest/reference/s3control/create-job.html) in the *AWS CLI Command Reference*. To use this example command, replace the `user input placeholders` with your own information. Replace *`IAM-role`* with the ARN of the IAM role that you created earlier. ``` aws s3control create-job \ --region us-west-2 \ --account-id acct-id \ --operation '{"S3PutObjectTagging": { "TagSet": [{"Key":"keyOne", "Value":"ValueOne"}] }}' \ --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::amzn-s3-demo-manifest-bucket/manifest.csv","ETag":"60e460c9d1046e73f7dde5043ac3ae85"}}' \ --report '{"Bucket":"arn:aws:s3:::amzn-s3-demo-completion-report-bucket","Prefix":"final-reports", "Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTasks"}' \ --priority 42 \ --role-arn IAM-role \ --client-request-token $(uuidgen) \ --description "job description" \ --no-confirmation-required ``` In response, Amazon S3 returns a job ID (for example, `00e123a4-c0d8-41f4-a0eb-b46f9ba5b07c`). You will need the job ID to identify, monitor, and modify the job. ------ #### [ Generate manifest ] The following example shows how to create an S3 Batch Operations `S3DeleteObjectTagging` job that automatically generates a manifest based on your object filter criteria. This criteria includes the creation date, key name, size, storage class, and tags. **To create a Batch Operations `S3DeleteObjectTagging` job by generating a manifest** 1. Use the following commands to create an AWS Identity and Access Management (IAM) role, and then create an IAM policy to assign permissions. The following role and policy grant Amazon S3 permission to delete object tags, which you will need when you create the job in a subsequent step. 1. Use the following example command to create an IAM role for Batch Operations to use. To use this example command, replace `S3BatchJobRole` with the name that you want to give to the role. ``` aws iam create-role \ --role-name S3BatchJobRole \ --assume-role-policy-document '{ "Version": "2012-10-17" , TCX5-2025-waiver;, "Statement":[ { "Effect":"Allow", "Principal":{ "Service":"batchoperations.s3.amazonaws.com" }, "Action":"sts:AssumeRole" } ] }' ``` Record the role's Amazon Resource Name (ARN). You will need the ARN when you create a job. 1. Use the following example command to create an IAM policy with the necessary permissions and attach it to the IAM role that you created in the previous step. For more information about the necessary permissions, see [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md). **Note** Batch Operations jobs that perform actions on directory buckets require specific permissions. For more information, see [AWS Identity and Access Management (IAM) for S3 Express One Zone](https://docs.aws.amazon.com//AmazonS3/latest/userguide/s3-express-security-iam.html). To use this example command, replace the `user input placeholders` as follows: + Replace `S3BatchJobRole` with the name of your IAM role. Make sure that this name matches the name that you used earlier. + Replace `DeleteObjectTaggingBatchJobPolicy` with the name that you want to give your IAM policy. + Replace `amzn-s3-demo-destination-bucket` with the name of the bucket that contains the objects that you want to apply tags to. + Replace `amzn-s3-demo-manifest-bucket` with the name of the bucket where you want to save the manifest. + Replace `amzn-s3-demo-completion-report-bucket` with the name of the bucket where you want the completion report to be delivered to. ``` aws iam put-role-policy \ --role-name S3BatchJobRole \ --policy-name DeleteObjectTaggingBatchJobPolicy \ --policy-document '{ "Version": "2012-10-17" , TCX5-2025-waiver;, "Statement":[ { "Effect":"Allow", "Action":[ "s3:DeleteObjectTagging", "s3:DeleteObjectVersionTagging" ], "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" }, { "Effect":"Allow", "Action":[ "s3:PutInventoryConfiguration" ], "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-manifest-bucket", "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] }, { "Effect":"Allow", "Action":[ "s3:PutObject", "s3:ListBucket" ], "Resource":[ "arn:aws:s3:::amzn-s3-demo-completion-report-bucket", "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*", "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] } ] }' ``` 1. Use the following example command to create the `S3DeleteObjectTagging` job. In this example, the values in the `--report` section specify the bucket, prefix, format, and scope of the job report that will be generated. The `--manifest-generator` section specifies information about the source bucket that contains the objects the job will act upon, information about the manifest output list that will be generated for the job, and filter criteria to narrow the scope of objects to be included in the manifest by creation date, name constraints, size, and storage class. The command also specifies the job's priority, IAM role, and AWS Region. For more information, see [https://docs.aws.amazon.com/cli/latest/reference/s3control/create-job.html](https://docs.aws.amazon.com/cli/latest/reference/s3control/create-job.html) in the *AWS CLI Command Reference*. To use this example command, replace the `user input placeholders` with your own information. Replace *`IAM-role`* with the ARN of the IAM role that you created earlier. ``` aws s3control create-job \ --account-id 012345678901 \ --operation '{ "S3DeleteObjectTagging": {} }' \ --report '{ "Bucket":"arn:aws:s3:::amzn-s3-demo-completion-report-bucket", "Prefix":"reports", "Format":"Report_CSV_20180820", "Enabled":true, "ReportScope":"AllTasks" }' \ --manifest-generator '{ "S3JobManifestGenerator": { "ExpectedBucketOwner": "012345678901", "SourceBucket": "arn:aws:s3:::amzn-s3-demo-source-bucket", "EnableManifestOutput": true, "ManifestOutputLocation": { "ExpectedManifestBucketOwner": "012345678901", "Bucket": "arn:aws:s3:::amzn-s3-demo-manifest-bucket", "ManifestPrefix": "prefix", "ManifestFormat": "S3InventoryReport_CSV_20211130" }, "Filter": { "CreatedAfter": "2023-09-01", "CreatedBefore": "2023-10-01", "KeyNameConstraint": { "MatchAnyPrefix": [ "prefix" ], "MatchAnySuffix": [ "suffix" ] }, "ObjectSizeGreaterThanBytes": 100, "ObjectSizeLessThanBytes": 200, "MatchAnyStorageClass": [ "STANDARD", "STANDARD_IA" ] } } }' \ --priority 2 \ --role-arn IAM-role \ --region us-east-1 ``` In response, Amazon S3 returns a job ID (for example, `00e123a4-c0d8-41f4-a0eb-b46f9ba5b07c`). You will need this job ID to identify, monitor, or modify the job. ------ ### Using the AWS SDK for Java To create your Batch Operations job with the AWS SDK for Java, you can choose between two approaches depending on whether you're specifying an existing manifest or generating a manifest automatically: + *Specify existing manifest:* Create an S3 Batch Operations job (such as `S3PutObjectTagging`) that acts on objects listed in an existing manifest file. This approach requires you to provide the manifest location, ETag, and format specifications. + *Generate manifest automatically:* Create an S3 Batch Operations job (such as `s3PutObjectCopy`) that automatically generates a manifest based on object filter criteria, including creation date, key name, and size constraints. Both approaches use the S3Control client to configure job operations, manifest specifications, job reports, IAM roles, and other job parameters including priority and confirmation requirements. For examples of how to create S3 Batch Operations jobs with the AWS SDK for Java, see [Create a batch job to copy objects](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_CreateJob_section.html) in the *Amazon S3 API Reference*. ### Using the REST API You can use the REST API to create a Batch Operations job. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html) in the *Amazon Simple Storage Service API Reference*. ## Job responses If the `CreateJob` request succeeds, Amazon S3 returns a job ID. The job ID is a unique identifier that Amazon S3 generates automatically so that you can identify your Batch Operations job and monitor its status. When you create a job through the AWS CLI, AWS SDKs, or REST API, you can set S3 Batch Operations to begin processing the job automatically. The job runs as soon as it's ready instead of waiting behind higher-priority jobs. When you create a job through the Amazon S3 console, you must review the job details and confirm that you want to run the job before Batch Operations can begin to process it. If a job remains in the suspended state for over 30 days, it will fail. # Operations supported by S3 Batch Operations You can use S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. S3 Batch Operations can perform a single operation on lists of Amazon S3 objects that you specify. A single job can perform a specified operation on billions of objects containing exabytes of data. Amazon S3 tracks progress, sends notifications, and stores a detailed completion report of all actions, providing a fully managed, auditable, and serverless experience. You can use S3 Batch Operations through the Amazon S3 console, AWS CLI, AWS SDKs, or Amazon S3 REST API. S3 Batch Operations supports the following operations: # Copy objects You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. The Batch Operations **Copy** operation copies each object that is specified in the manifest. You can copy objects to a bucket in the same AWS Region or to a bucket in a different Region. S3 Batch Operations supports most options available through Amazon S3 for copying objects. These options include setting object metadata, setting permissions, and changing an object's storage class. You can also use the **Copy** operation to copy existing unencrypted objects and write them back to the same bucket as encrypted objects. For more information, see [Encrypting objects with Amazon S3 Batch Operations](https://aws.amazon.com/blogs/storage/encrypting-objects-with-amazon-s3-batch-operations/). When you copy objects, you can change the checksum algorithm used to calculate the checksum of the object. If objects don't have an additional checksum calculated, you can also add one by specifying the checksum algorithm for Amazon S3 to use. For more information, see [Checking object integrity in Amazon S3](checking-object-integrity.md). For more information about copying objects in Amazon S3 and the required and optional parameters, see [Copying, moving, and renaming objects](copy-object.md) in this guide and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) in the *Amazon Simple Storage Service API Reference*. ## Restrictions and limitations When you're using the Batch Operations **Copy** operation, the following restrictions and limitations apply: + All source objects must be in one bucket. + All destination objects must be in one bucket. + You must have read permissions for the source bucket and write permissions for the destination bucket. + Objects to be copied can be up to 5 GB in size. + If you try to copy objects from the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive classes to the S3 Standard storage class, you must first restore these objects. For more information, see [Restoring an archived object](restoring-objects.md). + You must create your Batch Operations **Copy** jobs in the destination Region, which is the Region that you intend to copy the objects to. + All `CopyObject` options are supported except for conditional checks on entity tags (ETags) and server-side encryption with customer-provided encryption keys (SSE-C). + If the destination bucket is unversioned, you will overwrite any objects that have the same key names. + Objects aren't necessarily copied in the same order as they appear in the manifest. For versioned buckets, if preserving the current or noncurrent version order is important, copy all noncurrent versions first. Then, after the first job is complete, copy the current versions in a subsequent job. + Copying objects to the Reduced Redundancy Storage (RRS) class isn't supported. + A single Batch Operations Copy job can support a manifest with up to 20 billion objects. # Copying objects using S3 Batch Operations You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. You can use S3 Batch Operations to create a **Copy** (`CopyObject`) job to copy objects within the same account or to a different destination account. The following examples show how to store and use a manifest that is in a different account. The first example shows how you can use Amazon S3 Inventory to deliver the inventory report to the destination account for use during job creation. The second example shows how to use a comma-separated values (CSV) manifest in the source or destination account. The third example shows how to use the **Copy** operation to enable S3 Bucket Keys for existing objects that have been encrypted by using server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). **Topics** + [ # Using an inventory report to copy objects across AWS accounts ](specify-batchjob-manifest-xaccount-inventory.md) + [ # Using a CSV manifest to copy objects across AWS accounts ](specify-batchjob-manifest-xaccount-csv.md) + [ # Using Batch Operations to enable S3 Bucket Keys for SSE-KMS ](batch-ops-copy-example-bucket-key.md) # Using an inventory report to copy objects across AWS accounts You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. You can use S3 Batch Operations to create a **Copy** (`CopyObject`) job to copy objects within the same account or to a different destination account. You can use Amazon S3 Inventory to create an inventory report and use the report to create a list (manifest) of objects to copy with S3 Batch Operations. For more information about using a CSV manifest in the source or destination account, see [Using a CSV manifest to copy objects across AWS accounts](specify-batchjob-manifest-xaccount-csv.md). Amazon S3 Inventory generates inventories of the objects in a bucket. The resulting list is published to an output file. The bucket that is inventoried is called the source bucket, and the bucket where the inventory report file is stored is called the destination bucket. The Amazon S3 Inventory report can be configured to be delivered to another AWS account. Doing so allows S3 Batch Operations to read the inventory report when the job is created in the destination account. For more information about Amazon S3 Inventory source and destination buckets, see [Source and destination buckets](storage-inventory.md#storage-inventory-buckets). The easiest way to set up an inventory is by using the Amazon S3 console, but you can also use the Amazon S3 REST API, AWS Command Line Interface (AWS CLI), or AWS SDKs. The following console procedure contains the high-level steps for setting up permissions for an S3 Batch Operations job. In this procedure, you copy objects from a source account to a destination account, with the inventory report stored in the destination account. **To set up Amazon S3 Inventory for source and destination buckets owned by different accounts** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. Decide on (or create) a destination manifest bucket to store the inventory report in. In this procedure, the *destination account* is the account that owns both the destination manifest bucket and the bucket that the objects are copied to. 1. Configure an inventory report for a source bucket. For information about how to use the console to configure an inventory or how to encrypt an inventory list file, see [Configuring Amazon S3 Inventory](configure-inventory.md). When you configure the inventory report, you specify the destination bucket where you want the list to be stored. The inventory report for the source bucket is published to the destination bucket. In this procedure, the *source account* is the account that owns the source bucket. Choose **CSV** for the output format. When you enter information for the destination bucket, choose **Buckets in another account**. Then enter the name of the destination manifest bucket. Optionally, you can enter the account ID of the destination account. After the inventory configuration is saved, the console displays a message similar to the following: Amazon S3 could not create a bucket policy on the destination bucket. Ask the destination bucket owner to add the following bucket policy to allow Amazon S3 to place data in that bucket. The console then displays a bucket policy that you can use for the destination bucket. 1. Copy the destination bucket policy that appears on the console. 1. In the destination account, add the copied bucket policy to the destination manifest bucket where the inventory report is stored. 1. Create a role in the destination account that is based on the S3 Batch Operations trust policy. For more information about this trust policy, see [Trust policy](batch-ops-iam-role-policies.md#batch-ops-iam-role-policies-trust). For more information about creating a role, see [ Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*. Enter a name for the role (the following example role uses the name *`BatchOperationsDestinationRoleCOPY`*). Choose the **S3** service, and then choose the **S3 Batch Operations** use case, which applies the trust policy to the role. Then choose **Create policy** to attach the following policy to the role. To use this policy, replace the *`user input placeholders`* with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowBatchOperationsDestinationObjectCOPY", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectVersionAcl", "s3:PutObjectAcl", "s3:PutObjectVersionTagging", "s3:PutObjectTagging", "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectAcl", "s3:GetObjectTagging", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-destination-bucket/*", "arn:aws:s3:::amzn-s3-demo-source-bucket/*", "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] } ] } ``` ------ The role uses the policy to grant `batchoperations.s3.amazonaws.com` permission to read the manifest in the destination bucket. It also grants permissions to `GET` objects, access control lists (ACLs), tags, and versions in the source object bucket. And it grants permissions to `PUT` objects, ACLs, tags, and versions into the destination object bucket. 1. In the source account, create a bucket policy for the source bucket that grants the role that you created in the previous step permissions to `GET` objects, ACLs, tags, and versions in the source bucket. This step allows S3 Batch Operations to get objects from the source bucket through the trusted role. The following is an example of the bucket policy for the source account. To use this policy, replace the *`user input placeholders`* with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowBatchOperationsSourceObjectCOPY", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:role/BatchOperationsDestinationRoleCOPY" }, "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectAcl", "s3:GetObjectTagging", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging" ], "Resource": "arn:aws:s3:::amzn-s3-demo-source-bucket/*" } ] } ``` ------ 1. After the inventory report is available, create an S3 Batch Operations **Copy** (`CopyObject`) job in the destination account, and choose the inventory report from the destination manifest bucket. You need the ARN for the IAM role that you created in the destination account. For general information about creating a job, see [Creating an S3 Batch Operations job](batch-ops-create-job.md). For information about creating a job by using the console, see [Creating an S3 Batch Operations job](batch-ops-create-job.md). # Using a CSV manifest to copy objects across AWS accounts You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. You can use S3 Batch Operations to create a **Copy** (`CopyObject`) job to copy objects within the same account or to a different destination account. You can use a CSV manifest that's stored in the source account to copy objects across AWS accounts with S3 Batch Operations. To use an S3 Inventory report as a manifest, see [Using an inventory report to copy objects across AWS accounts](specify-batchjob-manifest-xaccount-inventory.md). For an example of the CSV format for manifest files, see [Creating a manifest file](batch-ops-create-job.md#create-manifest-file). The following procedure shows how to set up permissions when using an S3 Batch Operations job to copy objects from a source account to a destination account with a CSV manifest file that's stored in the source account. **To use a CSV manifest to copy objects across AWS accounts** 1. Create an AWS Identity and Access Management (IAM) role in the destination account that's based on the S3 Batch Operations trust policy. In this procedure, the *destination account* is the account that the objects are being copied to. For more information about the trust policy, see [Trust policy](batch-ops-iam-role-policies.md#batch-ops-iam-role-policies-trust). For more information about creating a role, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) in the *IAM User Guide*. If you create the role by using the console, enter a name for the role (the following example role uses the name `BatchOperationsDestinationRoleCOPY`). Choose the **S3** service, and then choose the **S3 Batch Operations** use case, which applies the trust policy to the role. Then choose **Create policy** to attach the following policy to the role. To use this policy, replace the *`user input placeholders`* with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowBatchOperationsDestinationObjectCOPY", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectVersionAcl", "s3:PutObjectAcl", "s3:PutObjectVersionTagging", "s3:PutObjectTagging", "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectAcl", "s3:GetObjectTagging", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-destination-bucket/*", "arn:aws:s3:::amzn-s3-demo-source-bucket/*", "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" ] } ] } ``` ------ Using the policy, the role grants `batchoperations.s3.amazonaws.com` permission to read the manifest in the source manifest bucket. It grants permissions to `GET` objects, access control lists (ACLs), tags, and versions in the source object bucket. It also grants permissions to `PUT` objects, ACLs, tags, and versions into the destination object bucket. 1. In the source account, create a bucket policy for the bucket that contains the manifest to grant the role that you created in the previous step permissions to `GET` objects and versions in the source manifest bucket. This step allows S3 Batch Operations to read the manifest by using the trusted role. Apply the bucket policy to the bucket that contains the manifest. The following is an example of the bucket policy to apply to the source manifest bucket. To use this policy, replace the *`user input placeholders`* with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowBatchOperationsSourceManifestRead", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::111122223333:user/ConsoleUserCreatingJob", "arn:aws:iam::111122223333:role/BatchOperationsDestinationRoleCOPY" ] }, "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::amzn-s3-demo-manifest-bucket/*" } ] } ``` ------ This policy also grants permissions to allow a console user who is creating a job in the destination account the same permissions in the source manifest bucket through the same bucket policy. 1. In the source account, create a bucket policy for the source bucket that grants the role that you created permissions to `GET` objects, ACLs, tags, and versions in the source object bucket. S3 Batch Operations can then get objects from the source bucket through the trusted role. The following is an example of the bucket policy for the bucket that contains the source objects. To use this policy, replace the *`user input placeholders`* with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowBatchOperationsSourceObjectCOPY", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:role/BatchOperationsDestinationRoleCOPY" }, "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectAcl", "s3:GetObjectTagging", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging" ], "Resource": "arn:aws:s3:::amzn-s3-demo-source-bucket/*" } ] } ``` ------ 1. Create an S3 Batch Operations job in the destination account. You need the Amazon Resource Name (ARN) for the role that you created in the destination account. For more information about creating a job, see [Creating an S3 Batch Operations job](batch-ops-create-job.md). # Using Batch Operations to enable S3 Bucket Keys for SSE-KMS S3 Bucket Keys reduce the cost of server-side encryption with AWS Key Management Service (AWS KMS) (SSE-KMS) by decreasing request traffic from Amazon S3 to AWS KMS. For more information, see [Reducing the cost of SSE-KMS with Amazon S3 Bucket Keys](bucket-key.md) and [Configuring your bucket to use an S3 Bucket Key with SSE-KMS for new objects](configuring-bucket-key.md). When you perform a `CopyObject` operation by using the REST API, AWS SDKs, or AWS CLI, you can enable or disable an S3 Bucket Key at the object level by adding the `x-amz-server-side-encryption-bucket-key-enabled` request header with a `true` or `false` value. When you configure an S3 Bucket Key for an object by using a `CopyObject` operation, Amazon S3 updates only the settings for that object. The S3 Bucket Key settings for the destination bucket don't change. If you submit a `CopyObject` request for an AWS KMS encrypted object to a bucket that has S3 Bucket Keys enabled, your object level operation will automatically use S3 Bucket Keys unless you disable the keys in the request header. If you don't specify an S3 Bucket Key for your object, Amazon S3 applies the S3 Bucket Key settings for the destination bucket to the object. To encrypt your existing Amazon S3 objects, you can use S3 Batch Operations. You can use the Batch Operations **Copy** operation to copy existing unencrypted objects and write them back to the same bucket as encrypted objects. For more information, see [Encrypting objects with Amazon S3 Batch Operations](https://aws.amazon.com/blogs/storage/encrypting-objects-with-amazon-s3-batch-operations/) on the AWS Storage Blog. In the following example, you use the Batch Operations **Copy** operation to enable S3 Bucket Keys on existing objects. For more information, see [Configuring an S3 Bucket Key at the object level](configuring-bucket-key-object.md). **Topics** + [ ## Considerations for using S3 Batch Operations to encrypt objects with S3 Bucket Keys enabled ](#bucket-key-ex-things-to-note) + [ ## Prerequisites ](#bucket-key-ex-prerequisites) + [ ## Step 1: Get your list of objects using Amazon S3 Inventory ](#bucket-key-ex-get-list-of-objects) + [ ## Step 2: Filter your object list with S3 Select ](#bucket-key-ex-filter-object-list-with-s3-select) + [ ## Step 3: Set up and run your S3 Batch Operations job ](#bucket-key-ex-setup-and-run-job) ## Considerations for using S3 Batch Operations to encrypt objects with S3 Bucket Keys enabled Consider the following issues when you use S3 Batch Operations to encrypt objects with S3 Bucket Keys enabled: + You will be charged for S3 Batch Operations jobs, objects, and requests in addition to any charges associated with the operation that S3 Batch Operations performs on your behalf, including data transfers, requests, and other charges. For more information, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing). + If you use a versioned bucket, each S3 Batch Operations job performed creates new encrypted versions of your objects. It also maintains the previous versions without an S3 Bucket Key configured. To delete the old versions, set up an S3 Lifecycle expiration policy for noncurrent versions as described in [Lifecycle configuration elements](intro-lifecycle-rules.md). + The copy operation creates new objects with new creation dates, which can affect lifecycle actions like archiving. If you copy all objects in your bucket, all the new copies have identical or similar creation dates. To further identify these objects and create different lifecycle rules for various data subsets, consider using object tags. ## Prerequisites Before you configure your objects to use an S3 Bucket Key, review [Changes to note before enabling an S3 Bucket Key](bucket-key.md#bucket-key-changes). To use this example, you must have an AWS account and at least one S3 bucket to hold your working files and encrypted results. You might also find much of the existing S3 Batch Operations documentation useful, including the following topics: + [S3 Batch Operations basics](batch-ops.md#batch-ops-basics) + [Creating an S3 Batch Operations job](batch-ops-create-job.md) + [Operations supported by S3 Batch Operations](batch-ops-operations.md) + [Managing S3 Batch Operations jobs](batch-ops-managing-jobs.md) ## Step 1: Get your list of objects using Amazon S3 Inventory To get started, identify the S3 bucket that contains the objects to encrypt, and get a list of its contents. An Amazon S3 Inventory report is the most convenient and affordable way to do this. The report provides the list of the objects in a bucket along with their associated metadata. In this step, the source bucket is the inventoried bucket, and the destination bucket is the bucket where you store the inventory report file. For more information about Amazon S3 Inventory source and destination buckets, see [Cataloging and analyzing your data with S3 Inventory](storage-inventory.md). The easiest way to set up an inventory is by using the AWS Management Console. But you can also use the REST API, AWS Command Line Interface (AWS CLI), or AWS SDKs. Before following these steps, be sure to sign in to the console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). If you encounter permission denied errors, add a bucket policy to your destination bucket. For more information, see [Grant permissions for S3 Inventory and S3 analytics](example-bucket-policies.md#example-bucket-policies-s3-inventory-1). **To get a list of objects using S3 Inventory** 1. Open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**, and choose a bucket that contains objects to encrypt. 1. On the **Management** tab, navigate to the **Inventory configurations** section, and choose **Create inventory configuration**. 1. Give your new inventory a name, enter the name of the destination S3 bucket, and optionally create a destination prefix for Amazon S3 to assign objects in that bucket. 1. For **Output format**, choose **CSV**. 1. (Optional) In the **Additional fields – *optional*** section, choose **Encryption** and any other report fields that interest you. Set the frequency for report deliveries to **Daily** so that the first report is delivered to your bucket sooner. 1. Choose **Create** to save your configuration. Amazon S3 can take up to 48 hours to deliver the first report, so check back when the first report arrives. After you receive your first report, proceed to the next step to filter your S3 Inventory report's contents. If you no longer want to receive inventory reports for this bucket, delete your S3 Inventory configuration. Otherwise, Amazon S3 continues to deliver reports on a daily or weekly schedule. An inventory list isn't a single point-in-time view of all objects. Inventory lists are a rolling snapshot of bucket items, which are eventually consistent (for example, the list might not include recently added or deleted objects). Combining S3 Inventory and S3 Batch Operations works best when you work with static objects, or with an object set that you created two or more days ago. To work with more recent data, use the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) (`GET` bucket) API operation to build your list of objects manually. If needed, repeat the process for the next few days or until your inventory report shows the desired status for all objects. ## Step 2: Filter your object list with S3 Select After you receive your S3 Inventory report, you can filter the report’s contents to list only the objects that aren't encrypted with S3 Bucket Keys enabled. If you want all your bucket’s objects encrypted with S3 Bucket Keys enabled, you can ignore this step. However, filtering your S3 Inventory report at this stage saves you the time and expense of re-encrypting objects that you previously encrypted with S3 Bucket Keys enabled. Although the following steps show how to filter by using [Amazon S3 Select](https://aws.amazon.com/blogs/aws/s3-glacier-select/), you can also use [Amazon Athena](https://aws.amazon.com/athena). To decide which tool to use, look at your S3 Inventory report’s `manifest.json` file. This file lists the number of data files that are associated with that report. If the number is large, use Amazon Athena because it runs across multiple S3 objects, whereas S3 Select works on one object at a time. For more information about using Amazon S3 and Athena together, see [Querying Amazon S3 Inventory with Amazon Athena](storage-inventory-athena-query.md) and "Using Athena" in the AWS Storage Blog post [Encrypting objects with Amazon S3 Batch Operations](https://aws.amazon.com/blogs/storage/encrypting-objects-with-amazon-s3-batch-operations). **To filter your S3 Inventory report by using S3 Select** 1. Open the `manifest.json` file from your inventory report and look at the `fileSchema` section of the JSON. This informs the query that you run on the data. The following JSON is an example `manifest.json` file for a CSV-formatted inventory on a bucket with versioning enabled. Depending on how you configured your inventory report, your manifest might look different. ``` { "sourceBucket": "batchoperationsdemo", "destinationBucket": "arn:aws:s3:::amzn-s3-demo-destination-bucket", "version": "2021-05-22", "creationTimestamp": "1558656000000", "fileFormat": "CSV", "fileSchema": "Bucket, Key, VersionId, IsLatest, IsDeleteMarker, BucketKeyStatus", "files": [ { "key": "demoinv/batchoperationsdemo/DemoInventory/data/009a40e4-f053-4c16-8c75-6100f8892202.csv.gz", "size": 72691, "MD5checksum": "c24c831717a099f0ebe4a9d1c5d3935c" } ] } ``` If versioning isn't activated on the bucket, or if you choose to run the report for the latest versions, the `fileSchema` is `Bucket`, `Key`, and `BucketKeyStatus`. If versioning *is* activated, depending on how you set up the inventory report, the `fileSchema` might include the following: `Bucket`, `Key`, `VersionId`, `IsLatest`, `IsDeleteMarker`, `BucketKeyStatus`. So pay attention to columns 1, 2, 3, and 6 when you run your query. S3 Batch Operations needs the bucket, key, and version ID as inputs to perform the job, in addition to the field to search by, which is `BucketKeyStatus`. You don't need the `VersionID` field, but it helps to specify the `VersionID` field when you operate on a versioned bucket. For more information, see [Working with objects in a versioning-enabled bucket](manage-objects-versioned-bucket.md). 1. Locate the data files for the inventory report. The `manifest.json` object lists the data files under **files**. 1. After you locate and select the data file in the S3 console, choose **Actions**, and then choose **Query with S3 Select**. 1. Keep the preset **CSV**, **Comma**, and **GZIP** fields selected, and choose **Next**. 1. To review your inventory report’s format before proceeding, choose **Show file preview**. 1. Enter the columns to reference in the SQL expression field, and choose **Run SQL**. The following expression returns columns 1–3 for all objects without an S3 Bucket Key configured. `select s._1, s._2, s._3 from s3object s where s._6 = 'DISABLED'` The following are example results. ``` batchoperationsdemo,0100059%7Ethumb.jpg,lsrtIxksLu0R0ZkYPL.LhgD5caTYn6vu batchoperationsdemo,0100074%7Ethumb.jpg,sd2M60g6Fdazoi6D5kNARIE7KzUibmHR batchoperationsdemo,0100075%7Ethumb.jpg,TLYESLnl1mXD5c4BwiOIinqFrktddkoL batchoperationsdemo,0200147%7Ethumb.jpg,amufzfMi_fEw0Rs99rxR_HrDFlE.l3Y0 batchoperationsdemo,0301420%7Ethumb.jpg,9qGU2SEscL.C.c_sK89trmXYIwooABSh batchoperationsdemo,0401524%7Ethumb.jpg,ORnEWNuB1QhHrrYAGFsZhbyvEYJ3DUor batchoperationsdemo,200907200065HQ%7Ethumb.jpg,d8LgvIVjbDR5mUVwW6pu9ahTfReyn5V4 batchoperationsdemo,200907200076HQ%7Ethumb.jpg,XUT25d7.gK40u_GmnupdaZg3BVx2jN40 batchoperationsdemo,201103190002HQ%7Ethumb.jpg,z.2sVRh0myqVi0BuIrngWlsRPQdb7qOS ``` 1. Download the results, save them into a CSV format, and upload them to Amazon S3 as your list of objects for the S3 Batch Operations job. 1. If you have multiple manifest files, run **Query with S3 Select** on those also. Depending on the size of the results, you could combine the lists and run a single S3 Batch Operations job or run each list as a separate job. To decide number of jobs to run, consider the [price](https://aws.amazon.com/s3/pricing/) of running each S3 Batch Operations job. ## Step 3: Set up and run your S3 Batch Operations job Now that you have your filtered CSV lists of S3 objects, you can begin the S3 Batch Operations job to encrypt the objects with S3 Bucket Keys enabled. A *job* refers collectively to the list (manifest) of objects provided, the operation performed, and the specified parameters. The easiest way to encrypt this set of objects with S3 Bucket Keys enabled is by using the **Copy** operation and specifying the same destination prefix as the objects listed in the manifest. In an unversioned bucket, this operation overwrites the existing objects. In a bucket with versioning turned on, this operation creates a newer, encrypted version of the objects. As part of copying the objects, specify that Amazon S3 should encrypt the objects with SSE-KMS encryption. This job copies the objects, so all of your objects will show an updated creation date upon completion, regardless of when you originally added them to Amazon S3. Also specify the other properties for your set of objects as part of the S3 Batch Operations job, including object tags and storage class. **Topics** + [ ### Set up your IAM policy ](#bucket-key-ex-set-up-iam-policy) + [ ### Set up your Batch Operations IAM role ](#bucket-key-ex-set-up-iam-role) + [ ### Enable S3 Bucket Keys for an existing bucket ](#bucket-key-ex-enable-s3-bucket-key-on-a-bucket) + [ ### Create your Batch Operations job ](#bucket-key-ex-create-job) + [ ### Run your Batch Operations job ](#bucket-key-ex-run-job) ### Set up your IAM policy 1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policy**, and then choose **Create policy**. 1. Choose the **JSON** tab. Choose **Edit policy** and add the example IAM policy that appears in the following code block. After copying the policy example into your [IAM Console](https://console.aws.amazon.com/iam/), replace the following: 1. Replace `amzn-s3-demo-source-bucket` with the name of your source bucket to copy objects from. 1. Replace `amzn-s3-demo-destination-bucket` with the name of your destination bucket to copy objects to. 1. Replace `amzn-s3-demo-manifest-bucket/manifest-key` with the name of your manifest object. 1. Replace `amzn-s3-demo-completion-report-bucket` with the name of the bucket where you want to save your completion reports. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "CopyObjectsToEncrypt", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectTagging", "s3:PutObjectAcl", "s3:PutObjectVersionTagging", "s3:PutObjectVersionAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectTagging", "s3:GetObjectVersion", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-source-bucket/*", "arn:aws:s3:::amzn-s3-demo-destination-bucket/*" ] }, { "Sid": "ReadManifest", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::amzn-s3-demo-manifest-bucket/manifest-key" }, { "Sid": "WriteReport", "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": "arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*" } ] } ``` ------ 1. Choose **Next: Tags**. 1. Add any tags that you want (optional), and choose **Next: Review**. 1. Add a policy name, optionally add a description, and choose **Create policy**. 1. Choose **Review policy** and **Save changes**. 1. With your S3 Batch Operations policy now complete, the console returns you to the IAM **Policies** page. Filter on the policy name, choose the button to the left of the policy name, choose **Policy actions**, and choose **Attach**. To attach the newly created policy to an IAM role, select the appropriate users, groups, or roles in your account and choose **Attach policy**. This takes you back to the IAM console. ### Set up your Batch Operations IAM role 1. On the [IAM Console](https://console.aws.amazon.com/iam/), in the navigation pane, choose **Roles**, and then choose **Create role**. 1. Choose **AWS service**, **S3**, and **S3 Batch Operations**. Then choose **Next: Permissions**. 1. Start entering the name of the IAM **policy** that you just created. Select the check box by the policy name when it appears, and choose **Next: Tags**. 1. (Optional) Add tags or keep the key and value fields blank for this exercise. Choose **Next: Review**. 1. Enter a role name, and accept the default description or add your own. Choose **Create role**. 1. Ensure that the user creating the job has the permissions in the following example. Replace `account-id` with your AWS account ID and `IAM-role-name` with the name that you plan to apply to the IAM role that you will create in the Batch Operations job creation step later. For more information, see [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md). ``` { "Sid": "AddIamPermissions", "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole" ], "Resource": "arn:aws:iam::account-id:role/IAM-role-name" } ``` ### Enable S3 Bucket Keys for an existing bucket 1. Open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the **Buckets** list, choose the bucket that you want to turn on an S3 Bucket Key for. 1. Choose **Properties**. 1. Under **Default encryption**, choose **Edit**. 1. Under **Encryption type**, you can choose between **Amazon S3 managed keys (SSE-S3)** and **AWS Key Management Service key (SSE-KMS)**. 1. If you chose **AWS Key Management Service key (SSE-KMS)**, under **AWS KMS key**, you can specify the AWS KMS key through one of the following options. + To choose from a list of available KMS keys, choose **Choose from your AWS KMS keys**. From the list of available keys, choose a symmetric encryption KMS key in the same Region as your bucket. Both the AWS managed key (`aws/s3`) and your customer managed keys appear in the list. + To enter the KMS key ARN, choose **Enter AWS KMS key ARN**, and then enter your KMS key ARN in the field that appears. + To create a new customer managed key in the AWS KMS console, choose **Create a KMS key**. 1. Under **Bucket Key**, choose **Enable**, and then choose **Save changes**. Now that an S3 Bucket Key is enabled at the bucket level, objects that are uploaded, modified, or copied into this bucket will inherit this encryption configuration by default. This includes objects that are copied by using Amazon S3 Batch Operations. ### Create your Batch Operations job 1. Open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the navigation pane, choose **Batch Operations**, and then choose **Create Job**. 1. Choose the **Region** where you store your objects, and choose **CSV** as the manifest type. 1. Enter the path or navigate to the CSV manifest file that you created earlier from S3 Select (or Athena) results. If your manifest contains version IDs, select that box. Choose **Next**. 1. Choose the **Copy** operation, and choose the copy destination bucket. You can keep server-side encryption disabled. As long as the bucket destination has S3 Bucket Keys enabled, the copy operation applies S3 Bucket Keys at the destination bucket. 1. (Optional) Choose a storage class and the other parameters as desired. The parameters that you specify in this step apply to all operations performed on the objects that are listed in the manifest. Choose **Next**. 1. To configure server-side encryption, follow these steps: 1. Under **Server-side encryption**, choose one of the following: + To keep the bucket settings for default server-side encryption of objects when storing them in Amazon S3, choose **Do not specify an encryption key**. As long as the bucket destination has S3 Bucket Keys enabled, the copy operation applies an S3 Bucket Key at the destination bucket. **Note** If the bucket policy for the specified destination requires objects to be encrypted before storing them in Amazon S3, you must specify an encryption key. Otherwise, copying objects to the destination will fail. + To encrypt objects before storing them in Amazon S3, choose **Specify an encryption key**. 1. Under **Encryption settings**, if you choose **Specify an encryption key**, you must choose either **Use destination bucket settings for default encryption** or **Override destination bucket settings for default encryption**. 1. If you choose **Override destination bucket settings for default encryption**, you must configure the following encryption settings. 1. Under **Encryption type**, you must choose either **Amazon S3 managed keys (SSE-S3)** or **AWS Key Management Service key (SSE-KMS)**. SSE-S3 uses one of the strongest block ciphers—256-bit Advanced Encryption Standard (AES-256) to encrypt each object. SSE-KMS provides you with more control over your key. For more information, see [Using server-side encryption with Amazon S3 managed keys (SSE-S3)](UsingServerSideEncryption.md) and [Using server-side encryption with AWS KMS keys (SSE-KMS)](UsingKMSEncryption.md). 1. If you choose **AWS Key Management Service key (SSE-KMS)**, under **AWS KMS key**, you can specify your AWS KMS key through one of the following options. + To choose from a list of available KMS keys, choose **Choose from your AWS KMS keys**, and then choose a symmetric encryption KMS key in the same Region as your bucket. Both the AWS managed key (`aws/s3`) and your customer managed keys appear in the list. + To enter the KMS key ARN, choose **Enter AWS KMS key ARN**, and enter your KMS key ARN in the field that appears. + To create a new customer managed key in the AWS KMS console, choose **Create a KMS key**. 1. Under **Bucket Key**, choose **Enable**. The copy operation applies an S3 Bucket Key at the destination bucket. 1. Give your job a description (or keep the default), set its priority level, choose a report type, and specify the **Path to completion report destination**. 1. In the **Permissions** section, be sure to choose the Batch Operations IAM role that you defined earlier. Choose **Next**. 1. Under **Review**, verify the settings. If you want to make changes, choose **Previous**. After confirming the Batch Operations settings, choose **Create job**. For more information, see [Creating an S3 Batch Operations job](batch-ops-create-job.md). ### Run your Batch Operations job The setup wizard automatically returns you to the S3 Batch Operations section of the Amazon S3 console. Your new job transitions from the **New** state to the **Preparing** state as S3 begins the process. During the Preparing state, S3 reads the job’s manifest, checks it for errors, and calculates the number of objects. 1. Choose the refresh button in the Amazon S3 console to check progress. Depending on the size of the manifest, reading can take minutes or hours. 1. After S3 finishes reading the job’s manifest, the job moves to the **Awaiting your confirmation** state. Choose the option button to the left of the Job ID, and choose **Run job**. 1. Check the settings for the job, and choose **Run job** in the bottom-right corner. After the job begins running, you can choose the refresh button to check progress through the console dashboard view or by selecting the specific job. 1. When the job is complete, you can view the **Successful** and **Failed** object counts to confirm that everything performed as expected. If you enabled job reports, check your job report for the exact cause of any failed operations. You can also perform these steps by using the AWS CLI, AWS SDKs, or Amazon S3 REST API. For more information about tracking job status and completion reports, see [Tracking job status and completion reports](batch-ops-job-status.md). For examples that show the copy operation with tags using the AWS CLI and AWS SDK for Java, see [Creating a Batch Operations job with job tags used for labeling](batch-ops-tags-create.md). # Compute checksums You can use S3 Batch Operations with the **Compute checksum** operation to perform checksum calculations for objects stored in Amazon S3 at rest. The **Compute checksum** operation calculates object checksums which you can use to validate data integrity without downloading or restoring objects for stored data. You can use the **Compute checksum** operation to calculate checksums for both composite and full object checksum types, for all supported checksum algorithms. With the **Compute checksum** operation, you can process billions of objects through a single job request. This batch operation is compatible with all S3 storage classes, regardless of object size. To create a **Compute checksum** job, use the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API. When you [enable server access logging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html), you can also receive log entries about your **Compute checksum** job. The **Compute checksum** job operation emits separate server access log events after completing the checksum computations. These log entries follow the standard [S3 server access logging format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html) and include fields such as operation type, timestamp, [error codes](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList), and the associated **Compute checksum** job ID. This logging provides an audit trail of checksum verification activities performed on your objects, helping you track and verify data integrity operations. **Note** The **Compute checksum** operation doesn’t support server-side encryption with customer-provided encryption keys (SSE-C) encrypted objects. However, you can use the **Compute checksum** operation with objects that are encrypted by using [server-side encryption with S3 managed keys (SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html), server-side encryption with AWS Key Management Service (DSSE-KMS). Make sure that you’ve [granted the proper AWS KMS permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html#require-sse-kms) to perform the **Compute checksum** operation. To get started with the **Compute checksum** operation using Batch Operations, you can either: + Manually create a new manifest file. + Use an existing manifest. + Direct Batch Operations to automatically generate a manifest based on object filter criteria that you [specify when you create your job](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html#specify-batchjob-manifest). Then, submit your **Compute checksum** job request and monitor its status. After the **Compute checksum** job finishes, you automatically receive a completion report in the specified destination bucket. This completion report contains checksum information for every object in the bucket, allowing you to verify data consistency. For more information about how to use this report to examine the job, see [Tracking job status and completion reports](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-job-status.html). For more information about **Compute checksum** capabilities and how to use **Compute checksum** in the console, see [Checking object integrity for data at rest in Amazon S3](checking-object-integrity-at-rest.md). For information about how to send REST requests to **Compute checksum**, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_DescribeJob.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_DescribeJob.html) and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html) in the *Amazon S3 API Reference*. The following sections explain how you can get started using the **Compute checksum** operation with S3 Batch Operations. **Topics** + [ ## S3 Batch Operations **Compute checksum** considerations ](#batch-ops-compute-checksum-considerations) + [ ## S3 Batch Operations completion report ](#batch-ops-compute-checksum-completion-report) ## S3 Batch Operations **Compute checksum** considerations Before using the **Compute checksum** operation, review the following list of considerations: + If your manifest includes a version ID field, you must provide a version ID for all objects in the manifest. If the version ID isn’t specified, the **Compute checksum** request performs the operation on the latest version of the object. + To receive **Compute checksum** operation details in your [server access logs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html), you must first [enable server access logging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html) on the source bucket and specify a destination bucket to store the logs. The destination bucket must also exist in the same AWS Region and AWS account as the source bucket. After configuring server access logging, the **Compute checksum** operation generates [log records](https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html#log-record-fields) that include standard fields such as operation type, HTTP status code, [S3 error codes](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList), timestamps, and the associated **Compute checksum** job ID. The **Compute checksum** operation runs asynchronously. As a result, the [log entries](https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html#log-record-fields) use a **Compute checksum** job ID, rather than a request ID, in its log entries. + The report generation can take up to a few hours for stored objects. + For the following S3 Glacier storage classes, the **Compute checksum** job can take up to a week to finish: + S3 Glacier Flexible Retrieval + S3 Glacier Deep Archive + For buckets where the completion report will be written, you must use the [bucket owner condition](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-owner-condition.html#bucket-owner-condition-when-to-use) when running the **Compute checksum** operation. If the actual bucket owner doesn't match the expected bucket owner for the submitted job request, then the job fails. For a list of S3 operations that don't support bucket owner condition, see [Restrictions and limitations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-owner-condition.html#bucket-owner-condition-restrictions-limitations). ## S3 Batch Operations completion report When you create a **Compute checksum** job, you can request an S3 Batch Operations completion report. This CSV file shows the objects, success or failure codes, outputs, and descriptions. For more information about job tracking and completion reports, see [Completion reports](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-job-status.html#batch-ops-completion-report). # Delete all object tags You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. The **Delete all object tags** operation removes all Amazon S3 object tag sets currently associated with the objects that are listed in the manifest. S3 Batch Operations doesn't support deleting tags from objects while keeping other tags in place. If the objects in your manifest are in a versioned bucket, you can remove the tag sets from a specific version of an object. To do so, you must specify a version ID for every object in the manifest. If you don't include a version ID for an object, S3 Batch Operations removes the tag set from the latest version of every object. For more information about Batch Operations manifests, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest). For more details about object tagging, see [Categorizing your objects using tags](object-tagging.md) in this guide, and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html), [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html), and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html) in the *Amazon Simple Storage Service API Reference*. **Warning** Running this job removes all object tag sets on every object listed in the manifest. To use the console to create a **Delete all object tags** job, see [Creating an S3 Batch Operations job](batch-ops-create-job.md). ## Restrictions and limitations When you're using Batch Operations to delete object tags, the following restrictions and limitations apply: + The AWS Identity and Access Management (IAM) role that you specify to run the job must have permissions to perform the underlying Amazon S3 `DeleteObjectTagging` operation. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html) in the *Amazon Simple Storage Service API Reference*. + S3 Batch Operations uses the Amazon S3 [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html) operation to remove the tag sets from every object in the manifest. All restrictions and limitations that apply to the underlying operation also apply to S3 Batch Operations jobs. + A single delete object tagging job can support a manifest with up to 20 billion objects. # Invoke AWS Lambda function You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. The **Invoke AWS Lambda function** Batch Operations operation initiates AWS Lambda functions to perform custom actions on objects that are listed in a manifest. This section describes how to create a Lambda function to use with S3 Batch Operations and how to create a job to invoke the function. The S3 Batch Operations job uses the `LambdaInvoke` operation to run a Lambda function on every object listed in a manifest. You can work with S3 Batch Operations by using the Amazon S3 console, AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. For more information about using Lambda, see [Getting Started with AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html) in the *AWS Lambda Developer Guide*. The following sections explain how you can get started using S3 Batch Operations with Lambda. **Topics** + [ ## Using Lambda with Batch Operations ](#batch-ops-invoke-lambda-using) + [ ## Creating a Lambda function to use with S3 Batch Operations ](#batch-ops-invoke-lambda-custom-functions) + [ ## Creating an S3 Batch Operations job that invokes a Lambda function ](#batch-ops-invoke-lambda-create-job) + [ ## Providing task-level information in Lambda manifests ](#storing-task-level-information-in-lambda) + [ ## S3 Batch Operations tutorial ](#batch-ops-tutorials-lambda) ## Using Lambda with Batch Operations When using S3 Batch Operations with AWS Lambda, you must create new Lambda functions specifically for use with S3 Batch Operations. You can't reuse existing Amazon S3 event-based functions with S3 Batch Operations. Event functions can only receive messages; they don't return messages. The Lambda functions that are used with S3 Batch Operations must accept and return messages. For more information about using Lambda with Amazon S3 events, see [Using AWS Lambda with Amazon S3](https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html) in the *AWS Lambda Developer Guide*. You create an S3 Batch Operations job that invokes your Lambda function. The job runs the same Lambda function on all of the objects listed in your manifest. You can control what versions of your Lambda function to use while processing the objects in your manifest. S3 Batch Operations support unqualified Amazon Resource Names (ARNs), aliases, and specific versions. For more information, see [ Introduction to AWS Lambda Versioning](https://docs.aws.amazon.com/lambda/latest/dg/versioning-intro.html) in the *AWS Lambda Developer Guide*. If you provide the S3 Batch Operations job with a function ARN that uses an alias or the `$LATEST` qualifier, and you update the version that either of those points to, S3 Batch Operations starts calling the new version of your Lambda function. This can be useful when you want to update functionality part of the way through a large job. If you don't want S3 Batch Operations to change the version that's used, provide the specific version in the `FunctionARN` parameter when you create your job. A single AWS Lambda job with S3 Batch Operations can support a manifest with up to 20 billion objects. ### Using Lambda and Batch Operations with directory buckets Directory buckets are a type of Amazon S3 bucket that's designed for workloads or performance-critical applications that require consistent single-digit millisecond latency. For more information, see [Directory buckets](https://docs.aws.amazon.com//AmazonS3/latest/userguide/directory-buckets-overview.html). There are special requirements for using Batch Operations to invoke Lambda functions that act on directory buckets. For example, you must structure your Lambda request using an updated JSON schema, and specify [https://docs.aws.amazon.com//AmazonS3/latest/API/API_control_LambdaInvokeOperation.html#AmazonS3-Type-control_LambdaInvokeOperation-InvocationSchemaVersion](https://docs.aws.amazon.com//AmazonS3/latest/API/API_control_LambdaInvokeOperation.html#AmazonS3-Type-control_LambdaInvokeOperation-InvocationSchemaVersion) 2.0 (not 1.0) when you create the job. This updated schema allows you to specify optional key-value pairs for [https://docs.aws.amazon.com//AmazonS3/latest/API/API_control_LambdaInvokeOperation.html#AmazonS3-Type-control_LambdaInvokeOperation-UserArguments](https://docs.aws.amazon.com//AmazonS3/latest/API/API_control_LambdaInvokeOperation.html#AmazonS3-Type-control_LambdaInvokeOperation-UserArguments), which you can use to modify certain parameters of existing Lambda functions. For more information, see [Automate object processing in Amazon S3 directory buckets with S3 Batch Operations and AWS Lambda](https://aws.amazon.com/blogs/storage/automate-object-processing-in-amazon-s3-directory-buckets-with-s3-batch-operations-and-aws-lambda/) in the AWS Storage Blog. ### Response and result codes S3 Batch Operations invokes the Lambda function with one or more keys, each of which has a `TaskID` associated with it. S3 Batch Operations expects a per-key result code from Lambda functions. Any task IDs sent in the request which aren't returned with a per-key result code will be given the result code from the `treatMissingKeysAs` field. `treatMissingKeysAs` is an optional request field and defaults to `TemporaryFailure`. The following table contains the other possible result codes and values for the `treatMissingKeysAs` field. | Response code | Description | | --- | --- | | Succeeded | The task completed normally. If you requested a job completion report, the task's result string is included in the report. | | TemporaryFailure | The task suffered a temporary failure and will be redriven before the job completes. The result string is ignored. If this is the final redrive, the error message is included in the final report. | | PermanentFailure | The task suffered a permanent failure. If you requested a job-completion report, the task is marked as Failed and includes the error message string. Result strings from failed tasks are ignored. | ## Creating a Lambda function to use with S3 Batch Operations This section provides example AWS Identity and Access Management (IAM) permissions that you must use with your Lambda function. It also contains an example Lambda function to use with S3 Batch Operations. If you have never created a Lambda function before, see [Tutorial: Using AWS Lambda with Amazon S3](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html) in the *AWS Lambda Developer Guide*. You must create Lambda functions specifically for use with S3 Batch Operations. You can't reuse existing Amazon S3 event-based Lambda functions because Lambda functions that are used for S3 Batch Operations must accept and return special data fields. **Important** AWS Lambda functions written in Java accept either [https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-core/src/main/java/com/amazonaws/services/lambda/runtime/RequestHandler.java](https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-core/src/main/java/com/amazonaws/services/lambda/runtime/RequestHandler.java) or [https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-core/src/main/java/com/amazonaws/services/lambda/runtime/RequestStreamHandler.java](https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-core/src/main/java/com/amazonaws/services/lambda/runtime/RequestStreamHandler.java) handler interfaces. However, to support S3 Batch Operations request and response format, AWS Lambda requires the `RequestStreamHandler` interface for custom serialization and deserialization of a request and response. This interface allows Lambda to pass an InputStream and OutputStream to the Java `handleRequest` method. Be sure to use the `RequestStreamHandler` interface when using Lambda functions with S3 Batch Operations. If you use a `RequestHandler` interface, the batch job will fail with "Invalid JSON returned in Lambda payload" in the completion report. For more information, see [Handler interfaces](https://docs.aws.amazon.com//lambda/latest/dg/java-handler.html#java-handler-interfaces) in the *AWS Lambda User Guide*. ### Example IAM permissions The following are examples of the IAM permissions that are necessary to use a Lambda function with S3 Batch Operations. **Example — S3 Batch Operations trust policy** The following is an example of the trust policy that you can use for the Batch Operations IAM role. This IAM role is specified when you create the job and gives Batch Operations permission to assume the IAM role. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "batchoperations.s3.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` **Example — Lambda IAM policy** The following is an example of an IAM policy that gives S3 Batch Operations permission to invoke the Lambda function and read the input manifest. **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "BatchOperationsLambdaPolicy", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:PutObject", "lambda:InvokeFunction" ], "Resource": "*" } ] } ``` ### Example request and response This section provides request and response examples for the Lambda function. **Example Request** The following is a JSON example of a request for the Lambda function. ``` { "invocationSchemaVersion": "1.0", "invocationId": "YXNkbGZqYWRmaiBhc2RmdW9hZHNmZGpmaGFzbGtkaGZza2RmaAo", "job": { "id": "f3cc4f60-61f6-4a2b-8a21-d07600c373ce" }, "tasks": [ { "taskId": "dGFza2lkZ29lc2hlcmUK", "s3Key": "customerImage1.jpg", "s3VersionId": "1", "s3BucketArn": "arn:aws:s3:us-east-1:0123456788:amzn-s3-demo-bucket1" } ] } ``` **Example Response** The following is a JSON example of a response for the Lambda function. ``` { "invocationSchemaVersion": "1.0", "treatMissingKeysAs" : "PermanentFailure", "invocationId" : "YXNkbGZqYWRmaiBhc2RmdW9hZHNmZGpmaGFzbGtkaGZza2RmaAo", "results": [ { "taskId": "dGFza2lkZ29lc2hlcmUK", "resultCode": "Succeeded", "resultString": "[\"Mary Major", \"John Stiles\"]" } ] } ``` ### Example Lambda function for S3 Batch Operations The following example Python Lambda removes a delete marker from a versioned object. As the example shows, keys from S3 Batch Operations are URL encoded. To use Amazon S3 with other AWS services, it's important that you URL decode the key that is passed from S3 Batch Operations. ``` import logging from urllib import parse import boto3 from botocore.exceptions import ClientError logger = logging.getLogger(__name__) logger.setLevel("INFO") s3 = boto3.client("s3") def lambda_handler(event, context): """ Removes a delete marker from the specified versioned object. :param event: The S3 batch event that contains the ID of the delete marker to remove. :param context: Context about the event. :return: A result structure that Amazon S3 uses to interpret the result of the operation. When the result code is TemporaryFailure, S3 retries the operation. """ # Parse job parameters from Amazon S3 batch operations invocation_id = event["invocationId"] invocation_schema_version = event["invocationSchemaVersion"] results = [] result_code = None result_string = None task = event["tasks"][0] task_id = task["taskId"] try: obj_key = parse.unquote_plus(task["s3Key"], encoding="utf-8") obj_version_id = task["s3VersionId"] bucket_name = task["s3BucketArn"].split(":")[-1] logger.info( "Got task: remove delete marker %s from object %s.", obj_version_id, obj_key ) try: # If this call does not raise an error, the object version is not a delete # marker and should not be deleted. response = s3.head_object( Bucket=bucket_name, Key=obj_key, VersionId=obj_version_id ) result_code = "PermanentFailure" result_string = ( f"Object {obj_key}, ID {obj_version_id} is not " f"a delete marker." ) logger.debug(response) logger.warning(result_string) except ClientError as error: delete_marker = error.response["ResponseMetadata"]["HTTPHeaders"].get( "x-amz-delete-marker", "false" ) if delete_marker == "true": logger.info( "Object %s, version %s is a delete marker.", obj_key, obj_version_id ) try: s3.delete_object( Bucket=bucket_name, Key=obj_key, VersionId=obj_version_id ) result_code = "Succeeded" result_string = ( f"Successfully removed delete marker " f"{obj_version_id} from object {obj_key}." ) logger.info(result_string) except ClientError as error: # Mark request timeout as a temporary failure so it will be retried. if error.response["Error"]["Code"] == "RequestTimeout": result_code = "TemporaryFailure" result_string = ( f"Attempt to remove delete marker from " f"object {obj_key} timed out." ) logger.info(result_string) else: raise else: raise ValueError( f"The x-amz-delete-marker header is either not " f"present or is not 'true'." ) except Exception as error: # Mark all other exceptions as permanent failures. result_code = "PermanentFailure" result_string = str(error) logger.exception(error) finally: results.append( { "taskId": task_id, "resultCode": result_code, "resultString": result_string, } ) return { "invocationSchemaVersion": invocation_schema_version, "treatMissingKeysAs": "PermanentFailure", "invocationId": invocation_id, "results": results, } ``` ## Creating an S3 Batch Operations job that invokes a Lambda function When creating an S3 Batch Operations job to invoke a Lambda function, you must provide the following: + The ARN of your Lambda function (which might include the function alias or a specific version number) + An IAM role with permission to invoke the function + The action parameter `LambdaInvokeFunction` For more information about creating an S3 Batch Operations job, see [Creating an S3 Batch Operations job](batch-ops-create-job.md) and [Operations supported by S3 Batch Operations](batch-ops-operations.md). The following example creates an S3 Batch Operations job that invokes a Lambda function by using the AWS CLI. To use this example, replace the *`user input placeholders`* with your own information. ``` aws s3control create-job --account-id account-id --operation '{"LambdaInvoke": { "FunctionArn": "arn:aws:lambda:region:account-id:function:LambdaFunctionName" } }' --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::amzn-s3-demo-manifest-bucket","ETag":"ManifestETag"}}' --report '{"Bucket":"arn:aws:s3:::amzn-s3-demo-bucket","Format":"Report_CSV_20180820","Enabled":true,"Prefix":"ReportPrefix","ReportScope":"AllTasks"}' --priority 2 --role-arn arn:aws:iam::account-id:role/BatchOperationsRole --region region --description "Lambda Function" ``` ## Providing task-level information in Lambda manifests When you use AWS Lambda functions with S3 Batch Operations, you might want additional data to accompany each task or key that's operated on. For example, you might want to have both a source object key and a new object key provided. Your Lambda function could then copy the source key to a new S3 bucket under a new name. By default, Batch Operations lets you specify only the destination bucket and a list of source keys in the input manifest to your job. The following examples describe how you can include additional data in your manifest so that you can run more complex Lambda functions. To specify per-key parameters in your S3 Batch Operations manifest to use in your Lambda function's code, use the following URL-encoded JSON format. The `key` field is passed to your Lambda function as if it were an Amazon S3 object key. But it can be interpreted by the Lambda function to contain other values or multiple keys, as shown in the following examples. **Note** The maximum number of characters for the `key` field in the manifest is 1,024. **Example — Manifest substituting the "Amazon S3 keys" with JSON strings** The URL-encoded version must be provided to S3 Batch Operations. ``` amzn-s3-demo-bucket,{"origKey": "object1key", "newKey": "newObject1Key"} amzn-s3-demo-bucket,{"origKey": "object2key", "newKey": "newObject2Key"} amzn-s3-demo-bucket,{"origKey": "object3key", "newKey": "newObject3Key"} ``` **Example — Manifest URL-encoded** This URL-encoded version must be provided to S3 Batch Operations. The non-URL-encoded version does not work. ``` amzn-s3-demo-bucket,%7B%22origKey%22%3A%20%22object1key%22%2C%20%22newKey%22%3A%20%22newObject1Key%22%7D amzn-s3-demo-bucket,%7B%22origKey%22%3A%20%22object2key%22%2C%20%22newKey%22%3A%20%22newObject2Key%22%7D amzn-s3-demo-bucket,%7B%22origKey%22%3A%20%22object3key%22%2C%20%22newKey%22%3A%20%22newObject3Key%22%7D ``` **Example — Lambda function with manifest format writing results to the job report** This URL-encoded manifest example contains pipe-delimited object keys for the following Lambda function to parse. ``` amzn-s3-demo-bucket,object1key%7Clower amzn-s3-demo-bucket,object2key%7Cupper amzn-s3-demo-bucket,object3key%7Creverse amzn-s3-demo-bucket,object4key%7Cdelete ``` This Lambda function shows how to parse a pipe-delimited task that's encoded into the S3 Batch Operations manifest. The task indicates which revision operation is applied to the specified object. ``` import logging from urllib import parse import boto3 from botocore.exceptions import ClientError logger = logging.getLogger(__name__) logger.setLevel("INFO") s3 = boto3.resource("s3") def lambda_handler(event, context): """ Applies the specified revision to the specified object. :param event: The Amazon S3 batch event that contains the ID of the object to revise and the revision type to apply. :param context: Context about the event. :return: A result structure that Amazon S3 uses to interpret the result of the operation. """ # Parse job parameters from Amazon S3 batch operations invocation_id = event["invocationId"] invocation_schema_version = event["invocationSchemaVersion"] results = [] result_code = None result_string = None task = event["tasks"][0] task_id = task["taskId"] # The revision type is packed with the object key as a pipe-delimited string. obj_key, revision = parse.unquote_plus(task["s3Key"], encoding="utf-8").split("|") bucket_name = task["s3BucketArn"].split(":")[-1] logger.info("Got task: apply revision %s to %s.", revision, obj_key) try: stanza_obj = s3.Bucket(bucket_name).Object(obj_key) stanza = stanza_obj.get()["Body"].read().decode("utf-8") if revision == "lower": stanza = stanza.lower() elif revision == "upper": stanza = stanza.upper() elif revision == "reverse": stanza = stanza[::-1] elif revision == "delete": pass else: raise TypeError(f"Can't handle revision type '{revision}'.") if revision == "delete": stanza_obj.delete() result_string = f"Deleted stanza {stanza_obj.key}." else: stanza_obj.put(Body=bytes(stanza, "utf-8")) result_string = ( f"Applied revision type '{revision}' to " f"stanza {stanza_obj.key}." ) logger.info(result_string) result_code = "Succeeded" except ClientError as error: if error.response["Error"]["Code"] == "NoSuchKey": result_code = "Succeeded" result_string = ( f"Stanza {obj_key} not found, assuming it was deleted " f"in an earlier revision." ) logger.info(result_string) else: result_code = "PermanentFailure" result_string = ( f"Got exception when applying revision type '{revision}' " f"to {obj_key}: {error}." ) logger.exception(result_string) finally: results.append( { "taskId": task_id, "resultCode": result_code, "resultString": result_string, } ) return { "invocationSchemaVersion": invocation_schema_version, "treatMissingKeysAs": "PermanentFailure", "invocationId": invocation_id, "results": results, } ``` ## S3 Batch Operations tutorial The following tutorial presents complete end-to-end procedures for some Batch Operations tasks with Lambda. In this tutorial, you learn how to set up Batch Operations to invoke a Lambda function for batch-transcoding of videos stored in an S3 source bucket. The Lambda function calls AWS Elemental MediaConvert to transcode the videos. + [Tutorial: Batch-transcoding videos with S3 Batch Operations](tutorial-s3-batchops-lambda-mediaconvert-video.md) # Replace all object tags You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. The **Replace all object tags** operation replaces the object tags on every object listed in the manifest. An object tag is a key-value pair of strings that you can use to store metadata about an object. To create a **Replace all object tags** job, you provide a set of tags that you want to apply. S3 Batch Operations applies the same set of tags to every object. The tag set that you provide replaces whatever tag sets are already associated with the objects in the manifest. S3 Batch Operations doesn't support adding tags to objects while leaving the existing tags in place. If the objects in your manifest are in a versioned bucket, you can apply the tag set to specific versions of every object. To do so, specify a version ID for every object in the manifest. If you don't include a version ID for any objects, S3 Batch Operations applies the tag set to the latest version of every object. For more information about Batch Operations manifests, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest). For more information about object tagging, see [Categorizing your objects using tags](object-tagging.md) in this guide, and see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html), [https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html), and [https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html) in the *Amazon Simple Storage Service API Reference*. To use the console to create a **Replace all object tags** job, see [Creating an S3 Batch Operations job](batch-ops-create-job.md). ## Restrictions and limitations When you're using Batch Operations to replace object tags, the following restrictions and limitations apply: + The AWS Identity and Access Management (IAM) role that you specify to run the Batch Operations job must have permissions to perform the underlying `PutObjectTagging` operation. For more information about the permissions required, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html) in the *Amazon Simple Storage Service API Reference*. + S3 Batch Operations uses the Amazon S3 [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html) operation to apply tags to each object in the manifest. All restrictions and limitations that apply to the underlying operation also apply to S3 Batch Operations jobs. + A single replace all object tags job can support a manifest with up to 20 billion objects. # Replace access control list (ACL) You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. The **Replace access control list (ACL)** operation replaces the access control lists (ACLs) for every object that's listed in the manifest. By using ACLs, you can define who can access an object and what actions they can perform. **Note** A majority of modern use cases in Amazon S3 no longer require the use of ACLs. We recommend that you keep ACLs disabled, except in circumstances where you need to control access for each object individually. With ACLs disabled, you can use policies to control access to all objects in your bucket, regardless of who uploaded the objects to your bucket. For more information, see [Controlling ownership of objects and disabling ACLs for your bucket](about-object-ownership.md). S3 Batch Operations support custom ACLs that you define and the canned ACLs that Amazon S3 provides with a predefined set of access permissions. If the objects in your manifest are in a versioned bucket, you can apply the ACLs to specific versions of every object. To do so, specify a version ID for every object in the manifest. If you don't include a version ID for any object, S3 Batch Operations applies the ACL to the latest version of the object. For more information about ACLs in Amazon S3, see [Access control list (ACL) overview](acl-overview.md). **S3 Block Public Access** If you want to limit public access to all objects in a bucket, we recommend using Amazon S3 Block Public Access instead of using S3 Batch Operations to apply ACLs. Block Public Access can limit public access on a per-bucket or account-wide basis with a single, simple operation that takes effect quickly. This behavior makes Amazon S3 Block Public Access a better choice when your goal is to control public access to all objects in a bucket or account. Use S3 Batch Operations only when you need to apply a customized ACL to every object in the manifest. For more information about S3 Block Public Access, see [Blocking public access to your Amazon S3 storage](access-control-block-public-access.md). **S3 Object Ownership** If the objects in the manifest are in a bucket that uses the **Bucket owner enforced** setting for Object Ownership, the **Replace access control list (ACL)** operation can only specify object ACLs that grant full control to the bucket owner. In this case, the **Replace access control list (ACL)** operation can't grant object ACL permissions to other AWS accounts or groups. For more information, see [Controlling ownership of objects and disabling ACLs for your bucket](about-object-ownership.md). ## Restrictions and limitations When you're using Batch Operations to replace ACLs, the following restrictions and limitations apply: + The AWS Identity and Access Management (IAM) role that you specify to run the **Replace access control list (ACL)** job must have permissions to perform the underlying Amazon S3 `PutObjectAcl` operation. For more information about the permissions required, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html) in the *Amazon Simple Storage Service API Reference*. + S3 Batch Operations uses the Amazon S3 `PutObjectAcl` operation to apply the specified ACL to every object in the manifest. Therefore, all restrictions and limitations that apply to the underlying `PutObjectAcl` operation also apply to S3 Batch Operations **Replace access control list (ACL)** jobs. + A single replace access control list job can support a manifest with up to 20 billion objects. # Restore objects with Batch Operations You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. The **Restore** operation initiates restore requests for the archived Amazon S3 objects that are listed in your manifest. The following archived objects must be restored before they can be accessed in real time: + Objects archived in the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes + Objects archived through the S3 Intelligent-Tiering storage class in the Archive Access or Deep Archive Access tiers Using a **Restore** ([https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_S3InitiateRestoreObjectOperation.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_S3InitiateRestoreObjectOperation.html)) operation in your S3 Batch Operations job results in a `RestoreObject` request for every object that's specified in the manifest. **Important** The **Restore** job only *initiates* the request to restore objects. S3 Batch Operations reports the job as complete for each object after the request is initiated for that object. Amazon S3 doesn't update the job or otherwise notify you when the objects have been restored. However, you can use S3 Event Notifications to receive notifications when the objects are available in Amazon S3. For more information, see [Amazon S3 Event Notifications](EventNotifications.md). When you create a **Restore** job, the following arguments are available: **ExpirationInDays** This argument specifies how long the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive object remains available in Amazon S3. **Restore** jobs that target S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects require that you set `ExpirationInDays` to `1` or greater. Don't set `ExpirationInDays` when creating **Restore** operation jobs that target S3 Intelligent-Tiering Archive Access and Deep Archive Access tier objects. Objects in S3 Intelligent-Tiering archive access tiers aren't subject to restore expiration, so specifying `ExpirationInDays` results in a `RestoreObject` request failure. **GlacierJobTier** Amazon S3 can restore objects by using one of three different retrieval tiers: `EXPEDITED`, `STANDARD`, and `BULK`. However, the S3 Batch Operations feature supports only the `STANDARD` and `BULK` retrieval tiers. For more information about the differences between the retrieval tiers, see [Understanding archive retrieval options](restoring-objects-retrieval-options.md). For more information about the pricing for each tier, see the **Requests & data retrievals** section on the [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/) page. ## Differences when restoring from S3 Glacier and S3 Intelligent-Tiering Restoring archived files from the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes differs from restoring files from the S3 Intelligent-Tiering storage class in the Archive Access or Deep Archive Access tiers. + When you restore from S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive, a temporary *copy* of the object is created. Amazon S3 deletes this copy after the value that you specified in the `ExpirationInDays` argument has elapsed. After this temporary copy is deleted, you must submit an additional restore request to access the object. + When restoring archived S3 Intelligent-Tiering objects, do *not* specify the `ExpirationInDays` argument. When you restore an object from the S3 Intelligent-Tiering Archive Access or Deep Archive Access tiers, the object transitions back into the S3 Intelligent-Tiering Frequent Access tier. After a minimum of 90 consecutive days of no access, the object automatically transitions into the Archive Access tier. After a minimum of 180 consecutive days of no access, the object automatically moves into the Deep Archive Access tier. + Batch Operations jobs can operate either on S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive storage class objects *or* on S3 Intelligent-Tiering Archive Access and Deep Archive Access storage tier objects. Batch Operations can't operate on both types of archived objects in the same job. To restore objects of both types, you *must* create separate Batch Operations jobs. ## Overlapping restores If your [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_S3InitiateRestoreObjectOperation.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_S3InitiateRestoreObjectOperation.html) job tries to restore an object that's already in the process of being restored, S3 Batch Operations proceeds as follows. The restore operation succeeds for the object if either of the following conditions is true: + Compared to the restoration request already in progress, this job's `ExpirationInDays` value is the same and its `GlacierJobTier` value is faster. + The previous restoration request has already been completed, and the object is currently available. In this case, Batch Operations updates the expiration date of the restored object to match the `ExpirationInDays` value that's specified in the in-progress restoration request. The restore operation fails for the object if any of the following conditions are true: + The restoration request already in progress hasn't yet been completed, and the restoration duration for this job (specified by the `ExpirationInDays` value) is different from the restoration duration that's specified in the in-progress restoration request. + The restoration tier for this job (specified by the `GlacierJobTier` value) is the same or slower than the restoration tier that's specified in the in-progress restoration request. ## Limitations `S3InitiateRestoreObjectOperation` jobs have the following limitations: + You must create the job in the same Region as the archived objects. + S3 Batch Operations doesn't support the `EXPEDITED` retrieval tier. + A single Batch Operations Restore job can support a manifest with up to 4 billion objects. For more information about restoring objects, see [Restoring an archived object](restoring-objects.md). # Update object encryption You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. The Batch Operations [https://docs.aws.amazon.com//AmazonS3/latest/API/API_control_UpdateObjectEncryptionOperation.html](https://docs.aws.amazon.com//AmazonS3/latest/API/API_control_UpdateObjectEncryptionOperation.html) operation updates the server-side encryption type of more than one Amazon S3 object with a single request. A single `UpdateObjectEncryption` operation job can support a manifest with up to 20 billion objects. The `UpdateObjectEncryption` operation is supported for all Amazon S3 storage classes that are supported by general purpose buckets. You can use the `UpdateObjectEncryption` operation to change encrypted objects from [ server-side encryption with Amazon S3 managed keys (SSE- S3)](https://docs.aws.amazon.com//AmazonS3/latest/userguide/UsingServerSideEncryption.html) to [AWS Key Management Service (AWS KMS) keys (SSE-KMS)](https://docs.aws.amazon.com//AmazonS3/latest/userguide/UsingKMSEncryption.html), or to apply S3 Bucket Keys. You can also use the `UpdateObjectEncryption` operation to change the customer-managed KMS key used to encrypt your data so that you can comply with custom key-rotation standards. When you create a Batch Operations job, you can generate an object list based on the source location and filter criteria that you specify. You can use the `MatchAnyObjectEncryption` filter to generate a list of objects from your bucket that you want to update and include in your manifest. The generated object list includes only source bucket objects with the indicated server-side encryption type. If you select SSE-KMS, you can optionally further filter your results by specifying a specific KMS Key ARN and Bucket Key enabled status. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestGeneratorFilter.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestGeneratorFilter.html) and [`SSEKMSFilter` in the *Amazon S3API Reference*](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_SSEKMSFilter.html). ## Restrictions and considerations When you're using the Batch Operations `UpdateObjectEncryption` operation, the following restrictions and considerations apply: + The `UpdateObjectEncryption` operation doesn't support objects that are unencrypted or objects that are encrypted with either dual-layer server-side encryption with AWS KMS keys (DSSE-KMS) or customer-provided encryption keys (SSE-C). Additionally, you cannot specify SSE-S3 encryption type `UpdateObjectEncryption` request. + You can use the `UpdateObjectEncryption` operation to update objects in buckets that have S3 Versioning enabled. To update the encryption type of a particular version, you must specify a version ID in your `UpdateObjectEncryption` request. If you don't specify version ID, the `UpdateObjectEncryption` request acts on the current version of the object. For more information about S3 Versioning, see [Retaining multiple versions of objects with S3 Versioning](Versioning.md). + The `UpdateObjectEncryption` operation fails on any object that has an S3 Object Lock retention mode or legal hold applied to it. If an object has a governance-mode retention period or a legal hold, you must first remove the Object Lock status on the object before you issue your `UpdateObjectEncryption` request. You can't use the `UpdateObjectEncryption` operation with objects that have an Object Lock compliance mode retention period applied to them. For more information about S3 Object Lock, see [Locking objects with Object Lock](object-lock.md). + `UpdateObjectEncryption` requests on source buckets with live replication enabled won't initiate replica events in the destination bucket. If you want to change the encryption type of objects in both your source and destination buckets, you must initiate separate `UpdateObjectEncryption` requests on the objects in the source and destination buckets. + By default, all `UpdateObjectEncryption` requests that specify a customer-managed KMS key are restricted to KMS keys that are owned by the bucket owner's AWS account. If you're using AWS Organizations, you can request the ability to use AWS KMS keys owned by other member accounts within your organization by contacting AWS Support. + If you use S3 Batch Replication to replicate datasets cross region and your objects previously had their server-side encryption type updated from SSE-S3 to SSE-KMS, you may need additional permissions. On the source region bucket, you must have `kms:decrypt` permissions. Then, you will need the `kms:decrypt` and `kms:encrypt` permissions for the bucket in the destination region. + Provide a full KMS key ARN in your `UpdateObjectEncryption` request. You can't use an alias name or alias ARN. You can determine the full KMS Key ARN in the AWS KMS Console or using the AWS KMS `DescribeKey` API. + To improve manifest generation performance when using the `KmsKeyArn` filter, use this filter in conjunction with other object metadata filters. For example, you can combine `KmsKeyArn` with `MatchAnyPrefix`, `CreatedAfter`, or `MatchAnyStorageClass` when you automatically generate a manifest in S3 Batch Operations. For more information about `UpdateObjectEncryption`, see [Updating server-side encryption for existing data](update-sse-encryption.md). ## Required permissions To perform the `UpdateObjectEncryption` operation, add the following AWS Identity and Access Management (IAM) policy to your IAM principal (user, role, or group). To use this policy, replace *`amzn-s3-demo-bucket`* with the name of the bucket that contains the objects that you want to update encryption for. Replace `amzn-s3-demo-manifest-bucket` with the name of the bucket that contains your manifest, and replace `amzn-s3-demo-completion-report-bucket` with the name of the bucket where you want to store your completion report. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "S3BatchOperationsUpdateEncryption", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:PutObject", "s3:UpdateObjectEncryption" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-target" "arn:aws:s3:::amzn-s3-demo-bucket-target-target/*" ] }, { "Sid": "S3BatchOperationsPolicyForManifestFile", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-manifest/*" ] }, { "Sid": "S3BatchOperationsPolicyForCompletionReport", "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-completion-report/*" ] }, { "Sid": "S3BatchOperationsPolicyManifestGeneration", "Effect": "Allow", "Action": [ "s3:PutInventoryConfiguration" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket-target" ] }, { "Sid": "AllowKMSOperationsForS3BatchOperations", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey", "kms:Encrypt", "kms:ReEncrypt*" ], "Resource": [ "arn:aws:kms:us-east-1:111122223333:key/01234567-89ab-cdef-0123-456789abcdef" ] } ] } ``` For the trust policy and permissions policy that you must attach to the IAM role that the S3 Batch Operations service principal assumes to run Batch Operations jobs on your behalf, see [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md) and [Update object encryption](batch-ops-iam-role-policies.md#batch-ops-update-encryption-policies). # Creating a Batch Operations job to update object encryption To update the server-side encryption type of more than one Amazon S3 object with a single request, you can use S3 Batch Operations. You can use S3 Batch Operations through the Amazon S3 console, AWS Command Line Interface (AWS CLI) AWS SDKs, or the Amazon S3 REST API. ## Using the AWS CLI To run the following commands, you must have the AWS CLI installed and configured. If you don’t have the AWS CLI installed, see [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com//cli/latest/userguide/getting-started-install.html) in the *AWS Command Line Interface User Guide*. Alternatively, you can run AWS CLI commands from the console by using AWS CloudShell. AWS CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the AWS Management Console. For more information, see [What is CloudShell?](https://docs.aws.amazon.com//cloudshell/latest/userguide/welcome.html) and [Getting started with AWS CloudShell](https://docs.aws.amazon.com//cloudshell/latest/userguide/getting-started.html) in the *AWS CloudShell User Guide*. **Example 1 – Create a Batch Operations job that updates encrypted objects from one AWS KMS key to another KMS key** The following example shows how to create an S3 Batch Operations job that updates the encryption settings for multiple objects in your general purpose bucket. This command creates a job that changes objects encrypted with one AWS Key Management Service (AWS KMS) key to use a different KMS key. This job also generates and saves a manifest of the affected objects and creates a report of the results. To use this command, replace the `user input placeholders` with your own information. ``` aws s3control create-job --account-id account-id \ --no-confirmation-required \ --operation '{"S3UpdateObjectEncryption": { "ObjectEncryption": { "SSEKMS": { "KMSKeyArn": "KMS-key-ARN-to-apply", "BucketKeyEnabled": false } } } }' \ --report '{ "Enabled": true, "Bucket": "report-bucket-ARN", "Format": "Report_CSV_20180820", "Prefix": "report", "ReportScope": "AllTasks" }' \ --manifest-generator '{ "S3JobManifestGenerator": { "ExpectedBucketOwner": "account-id", "SourceBucket": "source-bucket-ARN", "EnableManifestOutput": true, "ManifestOutputLocation": { "Bucket": "manifest-bucket-ARN", "ManifestFormat": "S3InventoryReport_CSV_20211130", "ManifestPrefix": "manifest-prefix" }, "Filter": { "MatchAnyObjectEncryption": [{ "SSEKMS": { "KmsKeyArn": "kms-key-ARN-to-match" } }] } } }' \ --priority 1 \ --role-arn batch-operations-role-ARN ``` For best performance, we recommend using the `KmsKeyArn` filter in conjunction with other object metadata filters, such as `MatchAnyPrefix`, `CreatedAfter`, or `MatchAnyStorageClass`. **Example 2 – Create a Batch Operations job that updates SSE-S3 encrypted objects to SSE-KMS** The following example shows how to create an S3 Batch Operations job that updates the encryption settings for multiple objects in your general purpose bucket. This command creates a job that changes objects encrypted by using server-side encryption with Amazon S3 managed keys (SSE-S3) to use server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS) instead. This job also generates and saves a manifest of the affected objects and creates a report of the results. To use this command, replace the `user input placeholders` with your own information. ``` aws s3control create-job --account-id account-id \ --no-confirmation-required \ --operation '{"S3UpdateObjectEncryption": { "ObjectEncryption": { "SSEKMS": { "KMSKeyArn": "KMS-key-ARN-to-apply", "BucketKeyEnabled": false } } } }' \ --report '{ "Enabled": true, "Bucket": "report-bucket-ARN", "Format": "Report_CSV_20180820", "Prefix": "report", "ReportScope": "AllTasks" }' \ --manifest-generator '{ "S3JobManifestGenerator": { "ExpectedBucketOwner": "account-id", "SourceBucket": "source-bucket-ARN", "EnableManifestOutput": true, "ManifestOutputLocation": { "Bucket": "manifest-bucket-ARN", "ManifestFormat": "S3InventoryReport_CSV_20211130", "ManifestPrefix": "manifest-prefix" }, "Filter": { "MatchAnyObjectEncryption": [{ "SSES3": {} }] } } }' \ --priority 1 \ --role-arn batch-operations-role-ARN ``` For best performance, we recommend using the `KmsKeyArn` filter in conjunction with other object metadata filters, such as `MatchAnyPrefix`, `CreatedAfter`, or `MatchAnyStorageClass`. # S3 Object Lock retention You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. You can use the **Object Lock retention** operation to apply retention dates for your objects by using either *governance* mode or *compliance* mode. These retention modes apply different levels of protection. You can apply either retention mode to any object version. Retention dates, like legal holds, prevent an object from being overwritten or deleted. Amazon S3 stores the *retain until date* specified in the object's metadata and protects the specified version of the object version until the retention period expires. You can use S3 Batch Operations with Object Lock to manage the retention dates of many Amazon S3 objects at once. You specify the list of target objects in your manifest and submit the manifest to Batch Operations for completion. For more information, see S3 Object Lock [Retention periods](object-lock.md#object-lock-retention-periods). Your S3 Batch Operations job with retention dates runs until completion, until cancellation, or until a failure state is reached. We recommend using S3 Batch Operations and S3 Object Lock retention when you want to add, change, or remove the retention date for many objects with a single request. Batch Operations verifies that Object Lock is enabled on your bucket before processing any keys in the manifest. To perform the operations and validation, Batch Operations needs the `s3:GetBucketObjectLockConfiguration` and `s3:PutObjectRetention` permissions in an AWS Identity and Access Management (IAM) role to allow Batch Operations to call Object Lock on your behalf. For more information, see [Object Lock considerations](object-lock-managing.md). For information about using this operation with the REST API, see `S3PutObjectRetention` in the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html) operation in the *Amazon Simple Storage Service API Reference*. For an AWS Command Line Interface (AWS CLI) example of using this operation, see [Using the AWS CLI](batch-ops-object-lock-retention.md#batch-ops-cli-object-lock-retention-example). For an AWS SDK for Java example, see [Using the AWS SDK for Java](batch-ops-object-lock-retention.md#batch-ops-examples-java-object-lock-retention). ## Restrictions and limitations When you're using Batch Operations to apply Object Lock retention periods, the following restrictions and limitations apply: + S3 Batch Operations doesn't make any bucket-level changes. + Versioning and S3 Object Lock must be configured on the bucket where the job is performed. + All objects listed in the manifest must be in the same bucket. + The operation works on the latest version of the object unless a version is explicitly specified in the manifest. + You need `s3:PutObjectRetention` permission in your IAM role to use an **Object Lock retention** job. + The `s3:GetBucketObjectLockConfiguration` IAM permission is required to confirm that Object Lock is enabled for the S3 bucket that you're performing the job on. + You can only extend the retention period of objects with `COMPLIANCE` mode retention dates applied, and this retention period can't be shortened. + A single S3 Object Lock retention job can support a manifest with up to 20 billion objects. # S3 Object Lock legal hold You can use Amazon S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. You can use the **Object Lock legal hold** operation to place a legal hold on an object version. Like setting a retention period, a legal hold prevents an object version from being overwritten or deleted. However, a legal hold doesn't have an associated retention period and remains in effect until it's removed. You can use S3 Batch Operations with Object Lock to add legal holds to many Amazon S3 objects at once. To do so, specify a list of the target objects in your manifest and submit that list to Batch Operations. Your S3 Batch Operations **Object Lock legal hold** job runs until completion, until cancellation, or until a failure state is reached. S3 Batch Operations verifies that Object Lock is enabled on your S3 bucket before processing any objects in the manifest. To perform the object operations and bucket-level validation, S3 Batch Operations needs the `s3:PutObjectLegalHold` and `s3:GetBucketObjectLockConfiguration` in an AWS Identity and Access Management (IAM) role. These permissions allow S3 Batch Operations to call S3 Object Lock on your behalf. When you create an S3 Batch Operations job to remove a legal hold, you only need to specify `Off` as the legal hold status. For more information, see [Object Lock considerations](object-lock-managing.md). For information about how to use this operation with the Amazon S3 REST API, see `S3PutObjectLegalHold` in the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html) operation in the *Amazon Simple Storage Service API Reference*. For an example of using this operation, see [Using the AWS SDK for Java](batch-ops-legal-hold-off.md#batch-ops-examples-java-object-lock-legalhold). ## Restrictions and limitations When you're using Batch Operations to apply or remove an Object Lock legal hold, the following restrictions and limitations apply: + S3 Batch Operations doesn't make any bucket-level changes. + All objects listed in the manifest must be in the same bucket. + Versioning and S3 Object Lock must be configured on the bucket where the job is performed. + The **Object Lock legal hold** operation works on the latest version of the object unless a version is explicitly specified in the manifest. + The `s3:PutObjectLegalHold` permission is required in your IAM role to add or remove a legal hold from objects. + The `s3:GetBucketObjectLockConfiguration` IAM permission is required to confirm that S3 Object Lock is enabled for the S3 bucket where the job is performed. + A single S3 Object Lock legal hold job can support a manifest with up to 20 billion objects. + [Copy objects](batch-ops-copy-object.md) + [Compute checksums](batch-ops-compute-checksums.md) + [Delete all object tags](batch-ops-delete-object-tagging.md) + [Invoke AWS Lambda function](batch-ops-invoke-lambda.md) + [Replace all object tags](batch-ops-put-object-tagging.md) + [Replace access control list (ACL)](batch-ops-put-object-acl.md) + [Restore objects with Batch Operations](batch-ops-initiate-restore-object.md) + [Update object encryption](batch-ops-update-encryption.md) + [Replicating existing objects with Batch Replication](s3-batch-replication-batch.md) + [S3 Object Lock retention](batch-ops-retention-date.md) + [S3 Object Lock legal hold](batch-ops-legal-hold.md) # Managing S3 Batch Operations jobs Amazon S3 provides a robust set of tools to help you manage your S3 Batch Operations jobs after you create them. This section describes the operations that you can use to manage and track your jobs by using the Amazon S3 console, AWS Command Line Interface (AWS CLI), AWS SDKs, or Amazon S3 REST API. **Topics** + [ ## Using the Amazon S3 console to manage your S3 Batch Operations jobs ](#batch-ops-manage-console) + [ # Listing jobs ](batch-ops-list-jobs.md) + [ # Viewing job details ](batch-ops-job-details.md) + [ # Assigning job priority ](batch-ops-job-priority.md) ## Using the Amazon S3 console to manage your S3 Batch Operations jobs Using the console, you can manage your S3 Batch Operations jobs. For example, you can: + View active and queued jobs + Check the status of a job + Change a job's priority + Confirm and run a job + Clone a job + Cancel a job **To manage Batch Operations using the console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Batch Operations**. 1. Choose the job that you want to manage. # Listing jobs You can retrieve a list of your S3 Batch Operations jobs. The list provides information about jobs that haven't yet finished, and jobs that finished within the last 90 days. For each job, the list includes details such as job ID, description, priority, current status, and the number of tasks that have succeeded and failed. You can filter your job list by status. If you retrieve the list by using the console, you can also search your jobs by description or ID and filter them by AWS Region. ## Get a list of `Active` and `Complete` jobs The following AWS CLI example gets a list of `Active` and `Complete` jobs. To use this example, replace the *`user input placeholders`* with your own information. ``` aws s3control list-jobs \ --region us-west-2 \ --account-id account-id \ --job-statuses '["Active","Complete"]' \ --max-results 20 ``` For more information and examples, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3control/list-jobs.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3control/list-jobs.html) in the *AWS CLI Command Reference*. # Viewing job details If you want more information about an Amazon S3 Batch Operations job than you can retrieve by listing jobs, you can view all the details for a single job. You can view details for jobs that haven't yet finished, or jobs that finished within the last 90 days. In addition to the information returned in a job list, a single job's details include information such as: + The operation parameters. + Details about the manifest. + Information about the completion report, if you configured one when you created the job. + The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role that you assigned to run the job. By viewing an individual job's details, you can access a job's entire configuration. To view a job’s details, you can use the Amazon S3 console or the AWS Command Line Interface (AWS CLI). ## Get an S3 Batch Operations job description in the Amazon S3 console **To view a Batch Operations job description by using the console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Batch Operations**. 1. Choose the job ID of the specific job to view its details. ## Get an S3 Batch Operations job description in the AWS CLI The following example gets the description of an S3 Batch Operations job by using the AWS CLI. To use the following example command, replace the *`user input placeholders`* with your own information. ``` aws s3control describe-job \ --region us-west-2 \ --account-id account-id \ --job-id 00e123a4-c0d8-41f4-a0eb-b46f9ba5b07c ``` For more information and examples, see [https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3control/describe-job.html](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3control/describe-job.html) in the *AWS CLI Command Reference*. # Assigning job priority You can assign each Amazon S3 Batch Operations job a numeric priority, which can be any positive integer. S3 Batch Operations prioritizes jobs according to the assigned priority. Jobs with a higher priority (or a higher numeric value for the priority parameter) are evaluated first. Priority is determined in descending order. For example, a job queue with a priority value of 10 is given scheduling preference over a job queue with a priority value of 1. You can change a job's priority while the job is running. If you submit a new job with a higher priority while a job is running, the lower-priority job can pause to allow the higher-priority job to run. Changing a job's priority doesn't affect the job's processing speed. **Note** S3 Batch Operations honors job priorities on a best-effort basis. Although jobs with higher priorities generally take precedence over jobs with lower priorities, Amazon S3 doesn't guarantee strict ordering of jobs. ## Using the S3 console **How to update job priority in the Amazon S3 console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Batch Operations**. 1. Select the specific job that you would like to manage. 1. Choose **Action**. In the dropdown list, choose **Update priority**. ## Using the AWS CLI The following example updates the job priority by using the AWS CLI. A higher number indicates a higher execution priority. To use the following example command, replace the *`user input placeholders`* with your own information. ``` aws s3control update-job-priority \ --region us-west-2 \ --account-id account-id \ --priority 98 \ --job-id 00e123a4-c0d8-41f4-a0eb-b46f9ba5b07c ``` ## Using the AWS SDK for Java To update the priority of an S3 Batch Operations job using the AWS SDK for Java, you can use the S3Control client to modify the job's execution priority, which determines the order in which jobs are processed relative to other jobs in the queue. For more information about job priority, see [Assigning job priority](#batch-ops-job-priority). For examples of how to update job priority with the AWS SDK for Java, see [Update the priority of a batch job](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_UpdateJobPriority_section.html) in the *Amazon S3 API Reference*. # Tracking job status and completion reports With S3 Batch Operations, you can view and update job status, add notifications and logging, track job failures, and generate completion reports. **Topics** + [ ## Job statuses ](#batch-ops-job-status-table) + [ ## Updating job status ](#updating-job-statuses) + [ ## Notifications and logging ](#batch-ops-notifications) + [ ## Tracking job failures ](#batch-ops-job-status-failure) + [ ## Completion reports ](#batch-ops-completion-report) + [ # Examples: Tracking an S3 Batch Operations job in Amazon EventBridge through AWS CloudTrail ](batch-ops-examples-event-bridge-cloud-trail.md) + [ # Examples: S3 Batch Operations completion reports ](batch-ops-examples-reports.md) ## Job statuses After you create and run a job, it progresses through a series of statuses. The following table describes the statuses and possible transitions between them. | Status | Description | Transitions | | --- | --- | --- | | `New` | A job begins in the `New` state when you create it. | A job automatically moves to the `Preparing` state when Amazon S3 begins processing the manifest object. | | `Preparing` | Amazon S3 is processing the manifest object and other job parameters to set up and run the job. | A job automatically moves to the `Ready` state after Amazon S3 finishes processing the manifest and other parameters. The job is then ready to begin running the specified operation on the objects listed in the manifest.If the job requires confirmation before running, such as when you create a job using the Amazon S3 console, then the job transitions from `Preparing` to `Suspended`. It remains in the `Suspended` state until you confirm that you want to run it. | | `Suspended` | The job requires confirmation, but you haven't yet confirmed that you want to run it. Only jobs that you create using the Amazon S3 console require confirmation. A job that's created using the console enters the `Suspended` state immediately after `Preparing`. After you confirm that you want to run the job and the job becomes `Ready`, it never returns to the `Suspended` state. | After you confirm that you want to run the job, its status changes to `Ready`. | | `Ready` | Amazon S3 is ready to begin running the requested object operations. | A job automatically moves to `Active` when Amazon S3 begins to run it. The amount of time that a job remains in the `Ready` state depends on whether you have higher-priority jobs running already and how long those jobs take to complete. | | `Active` | Amazon S3 is performing the requested operation on the objects listed in the manifest. While a job is `Active`, you can monitor its progress using the Amazon S3 console or the `DescribeJob` operation through the REST API, AWS CLI, or AWS SDKs. | A job moves out of the `Active` state when the job is no longer running operations on objects. This behavior can happen automatically, such as when a job completes successfully or fails. Or this behavior can occur as a result of user actions, such as canceling a job. The state that the job moves to depends on the reason for the transition. | | `Pausing` | The job is transitioning to `Paused` from another state. | A job automatically moves to `Paused` when the `Pausing` stage is finished. | | `Paused` | A job can become `Paused` if you submit another job with a higher priority while the current job is running. | A `Paused` job automatically returns to `Active` after any higher-priority jobs that are blocking the job's' execution complete, fail, or are suspended. | | `Completing` | The job is transitioning to `Complete` from another state. | A job automatically moves to `Complete` when the `Completing` stage is finished. | | `Complete` | The job has finished performing the requested operation on all objects in the manifest. The operation might have succeeded or failed for every object. If you configured the job to generate a completion report, the report is available as soon as the job is `Complete`. | `Complete` is a terminal state. Once a job reaches `Complete`, it doesn't transition to any other state. | | `Cancelling` | The job is transitioning to the `Cancelled` state. | A job automatically moves to `Cancelled` when the `Cancelling` stage is finished. | | `Cancelled` | You requested that the job be canceled, and S3 Batch Operations has successfully canceled the job. The job won't submit any new requests to Amazon S3. | `Cancelled` is a terminal state. After a job reaches `Cancelled`, the job won't transition to any other state. | | `Failing` | The job is transitioning to the `Failed` state. | A job automatically moves to `Failed` once the `Failing` stage is finished. | | `Failed` | The job has failed and is no longer running. For more information about job failures, see [Tracking job failures](#batch-ops-job-status-failure). | `Failed` is a terminal state. After a job reaches `Failed`, it won't transition to any other state. | ## Updating job status The following AWS CLI and AWS SDK for Java examples update the status of a Batch Operations job. For more information about using the Amazon S3 console to manage Batch Operations jobs, see [Using the Amazon S3 console to manage your S3 Batch Operations jobs](batch-ops-managing-jobs.md#batch-ops-manage-console). ### Using the AWS CLI To use the following example commands, replace the *`user input placeholders`* with your own information. + If you didn't specify the `--no-confirmation-required` parameter in your `create-job` command, the job remains in a suspended state until you confirm the job by setting its status to `Ready`. Amazon S3 then makes the job eligible for execution. ``` aws s3control update-job-status \ --region us-west-2 \ --account-id 123456789012 \ --job-id 00e123a4-c0d8-41f4-a0eb-b46f9ba5b07c \ --requested-job-status 'Ready' ``` + Cancel the job by setting the job status to `Cancelled`. ``` aws s3control update-job-status \ --region us-west-2 \ --account-id 123456789012 \ --job-id 00e123a4-c0d8-41f4-a0eb-b46f9ba5b07c \ --status-update-reason "No longer needed" \ --requested-job-status Cancelled ``` ### Using the AWS SDK for Java For examples of how to update job status with the AWS SDK for Java, see [Update the status of a batch job](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_UpdateJobStatus_section.html) in the *Amazon S3 API Reference*. ## Notifications and logging In addition to requesting completion reports, you can also capture, review, and audit Batch Operations activity by using AWS CloudTrail. Because Batch Operations uses existing Amazon S3 API operations to perform tasks, those tasks also emit the same events that they would if you called them directly. Therefore, you can track and record the progress of your job and all of its tasks by using the same notification, logging, and auditing tools and processes that you already use with Amazon S3. For more information, see the examples in the following sections. **Note** Batch Operations generates both management and data events in CloudTrail during job execution. The volume of these events scale with the number of keys in each job's manifest. For more information, see the [CloudTrail pricing](https://aws.amazon.com/cloudtrail/pricing/) page, which includes examples of how pricing changes depending on the number of trails that you have configured in your account. To learn how to configure and log events to fit your needs, see [Create your first trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-tutorial.html#tutorial-step2) in the *AWS CloudTrail User Guide*. For more information about Amazon S3 events, see [Amazon S3 Event Notifications](EventNotifications.md). ## Tracking job failures If an S3 Batch Operations job encounters a problem that prevents it from running successfully, such as not being able to read the specified manifest, the job fails. When a job fails, it generates one or more failure codes or failure reasons. S3 Batch Operations stores the failure codes and reasons with the job so that you can view them by requesting the job's details. If you requested a completion report for the job, the failure codes and reasons also appear there. To prevent jobs from running a large number of unsuccessful operations, Amazon S3 imposes a task-failure threshold on every Batch Operations job. When a job has run at least 1,000 tasks, Amazon S3 monitors the task-failure rate. At any point, if the failure rate (the number of tasks that have failed as a proportion of the total number of tasks that have run) exceeds 50 percent, the job fails. If your job fails because it exceeded the task-failure threshold, you can identify the cause of the failures. For example, you might have accidentally included some objects in the manifest that don't exist in the specified bucket. After fixing the errors, you can resubmit the job. **Note** S3 Batch Operations operates asynchronously and the tasks don't necessarily run in the order that the objects are listed in the manifest. Therefore, you can't use the manifest ordering to determine which objects' tasks succeeded and which ones failed. Instead, you can examine the job's completion report (if you requested one) or view your AWS CloudTrail event logs to help determine the source of the failures. ## Completion reports When you create a job, you can request a completion report. As long as S3 Batch Operations successfully invokes at least one task, Amazon S3 generates a completion report after the job finishes running tasks, fails, or is canceled. You can configure the completion report to include all tasks or only failed tasks. The completion report includes the job configuration, status, and information for each task, including the object key and version, status, error codes, and descriptions of any errors. Completion reports provide an easy way to view the results of your tasks in a consolidated format with no additional setup required. Completion reports are encrypted by using server-side encryption with Amazon S3 managed keys (SSE-S3). For an example of a completion report, see [Examples: S3 Batch Operations completion reports](batch-ops-examples-reports.md). If you don't configure a completion report, you can still monitor and audit your job and its tasks by using CloudTrail and Amazon CloudWatch. For more information, see the following topics: **Topics** + [ ## Job statuses ](#batch-ops-job-status-table) + [ ## Updating job status ](#updating-job-statuses) + [ ## Notifications and logging ](#batch-ops-notifications) + [ ## Tracking job failures ](#batch-ops-job-status-failure) + [ ## Completion reports ](#batch-ops-completion-report) + [ # Examples: Tracking an S3 Batch Operations job in Amazon EventBridge through AWS CloudTrail ](batch-ops-examples-event-bridge-cloud-trail.md) + [ # Examples: S3 Batch Operations completion reports ](batch-ops-examples-reports.md) # Examples: Tracking an S3 Batch Operations job in Amazon EventBridge through AWS CloudTrail Amazon S3 Batch Operations job activity is recorded as events in AWS CloudTrail. You can create a custom rule in Amazon EventBridge and send these events to the target notification resource of your choice, such as Amazon Simple Notification Service (Amazon SNS). **Note** Amazon EventBridge is the preferred way to manage your events. Amazon CloudWatch Events and EventBridge are the same underlying service and API, but EventBridge provides more features. Changes that you make in either CloudWatch or EventBridge appear in each console. For more information, see the *[Amazon EventBridge User Guide](https://docs.aws.amazon.com/eventbridge/latest/userguide/)*. **Topics** + [ ## S3 Batch Operations events recorded in CloudTrail ](#batch-ops-examples-cloud-trail-events) + [ ## EventBridge rule for tracking S3 Batch Operations job events ](#batch-ops-examples-event-bridge) ## S3 Batch Operations events recorded in CloudTrail When a Batch Operations job is created, it is recorded as a `JobCreated` event in CloudTrail. As the job runs, it changes state during processing, and other `JobStatusChanged` events are recorded in CloudTrail. You can view these events on the [CloudTrail console](https://console.aws.amazon.com/cloudtrail). For more information about CloudTrail, see the [https://docs.aws.amazon.com/awscloudtrail/latest/userguide/how-cloudtrail-works.html](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/how-cloudtrail-works.html). **Note** Only S3 Batch Operations job `status-change` events are recorded in CloudTrail. **Example — S3 Batch Operations job completion event recorded by CloudTrail** ``` { "eventVersion": "1.05", "userIdentity": { "accountId": "123456789012", "invokedBy": "s3.amazonaws.com" }, "eventTime": "2020-02-05T18:25:30Z", "eventSource": "s3.amazonaws.com", "eventName": "JobStatusChanged", "awsRegion": "us-west-2", "sourceIPAddress": "s3.amazonaws.com", "userAgent": "s3.amazonaws.com", "requestParameters": null, "responseElements": null, "eventID": "f907577b-bf3d-4c53-b9ed-8a83a118a554", "readOnly": false, "eventType": "AwsServiceEvent", "recipientAccountId": "123412341234", "serviceEventDetails": { "jobId": "d6e58ec4-897a-4b6d-975f-10d7f0fb63ce", "jobArn": "arn:aws:s3:us-west-2:181572960644:job/d6e58ec4-897a-4b6d-975f-10d7f0fb63ce", "status": "Complete", "jobEventId": "b268784cf0a66749f1a05bce259804f5", "failureCodes": [], "statusChangeReason": [] } } ``` ## EventBridge rule for tracking S3 Batch Operations job events The following example shows how to create a rule in Amazon EventBridge to capture S3 Batch Operations events recorded by AWS CloudTrail to a target of your choice. To do this, you create a rule by following all the steps in [Creating EventBridge rules that react to events](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html). You paste the following S3 Batch Operations custom event pattern policy where applicable, and choose the target service of your choice. **S3 Batch Operations custom event pattern policy** ``` { "source": [ "aws.s3" ], "detail-type": [ "AWS Service Event via CloudTrail" ], "detail": { "eventSource": [ "s3.amazonaws.com" ], "eventName": [ "JobCreated", "JobStatusChanged" ] } } ``` The following examples are two Batch Operations events that were sent to Amazon Simple Queue Service (Amazon SQS) from an EventBridge event rule. A Batch Operations job goes through many different states while processing (`New`, `Preparing`, `Active`, etc.), so you can expect to receive several messages for each job. **Example — JobCreated sample event** ``` { "version": "0", "id": "51dc8145-541c-5518-2349-56d7dffdf2d8", "detail-type": "AWS Service Event via CloudTrail", "source": "aws.s3", "account": "123456789012", "time": "2020-02-27T15:25:49Z", "region": "us-east-1", "resources": [], "detail": { "eventVersion": "1.05", "userIdentity": { "accountId": "11112223334444", "invokedBy": "s3.amazonaws.com" }, "eventTime": "2020-02-27T15:25:49Z", "eventSource": "s3.amazonaws.com", "eventName": "JobCreated", "awsRegion": "us-east-1", "sourceIPAddress": "s3.amazonaws.com", "userAgent": "s3.amazonaws.com", "eventID": "7c38220f-f80b-4239-8b78-2ed867b7d3fa", "readOnly": false, "eventType": "AwsServiceEvent", "serviceEventDetails": { "jobId": "e849b567-5232-44be-9a0c-40988f14e80c", "jobArn": "arn:aws:s3:us-east-1:181572960644:job/e849b567-5232-44be-9a0c-40988f14e80c", "status": "New", "jobEventId": "f177ff24f1f097b69768e327038f30ac", "failureCodes": [], "statusChangeReason": [] } } } ``` **Example — JobStatusChanged job completion event** ``` { "version": "0", "id": "c8791abf-2af8-c754-0435-fd869ce25233", "detail-type": "AWS Service Event via CloudTrail", "source": "aws.s3", "account": "123456789012", "time": "2020-02-27T15:26:42Z", "region": "us-east-1", "resources": [], "detail": { "eventVersion": "1.05", "userIdentity": { "accountId": "1111222233334444", "invokedBy": "s3.amazonaws.com" }, "eventTime": "2020-02-27T15:26:42Z", "eventSource": "s3.amazonaws.com", "eventName": "JobStatusChanged", "awsRegion": "us-east-1", "sourceIPAddress": "s3.amazonaws.com", "userAgent": "s3.amazonaws.com", "eventID": "0238c1f7-c2b0-440b-8dbd-1ed5e5833afb", "readOnly": false, "eventType": "AwsServiceEvent", "serviceEventDetails": { "jobId": "e849b567-5232-44be-9a0c-40988f14e80c", "jobArn": "arn:aws:s3:us-east-1:181572960644:job/e849b567-5232-44be-9a0c-40988f14e80c", "status": "Complete", "jobEventId": "51f5ac17dba408301d56cd1b2c8d1e9e", "failureCodes": [], "statusChangeReason": [] } } } ``` # Examples: S3 Batch Operations completion reports When you create an S3 Batch Operations job, you can request a completion report for all tasks or just for failed tasks. As long as at least one task has been invoked successfully, S3 Batch Operations generates a report for jobs that have completed, failed, or been canceled. The completion report contains additional information for each task, including the object key name and version, status, error codes, and descriptions of any errors. The description of errors for each failed task can be used to diagnose issues that occur during job creation, such as permissions. For **Compute checksum** jobs, the completion report contains checksum values for every object. **Note** Completion reports are always encrypted with Amazon S3 managed keys (SSE-S3). **Example — Top-level manifest result file** The top-level `manifest.json` file contains the locations of each succeeded report and (if the job had any failures) the location of failed reports, as shown in the following example. ``` { "Format": "Report_CSV_20180820", "ReportCreationDate": "2019-04-05T17:48:39.725Z", "Results": [ { "TaskExecutionStatus": "succeeded", "Bucket": "my-job-reports", "MD5Checksum": "83b1c4cbe93fc893f54053697e10fd6e", "Key": "job-f8fb9d89-a3aa-461d-bddc-ea6a1b131955/results/6217b0fab0de85c408b4be96aeaca9b195a7daa5.csv" }, { "TaskExecutionStatus": "failed", "Bucket": "my-job-reports", "MD5Checksum": "22ee037f3515975f7719699e5c416eaa", "Key": "job-f8fb9d89-a3aa-461d-bddc-ea6a1b131955/results/b2ddad417e94331e9f37b44f1faf8c7ed5873f2e.csv" } ], "ReportSchema": "Bucket, Key, VersionId, TaskStatus, ErrorCode, HTTPStatusCode, ResultMessage" } ``` **Succeeded task reports** Succeeded tasks reports contain the following for the *successful* tasks: + `Bucket` + `Key` + `VersionId` + `TaskStatus` + `ErrorCode` + `HTTPStatusCode` + `ResultMessage` **Failed task reports** Failed task reports contain the following information for all *failed* tasks: + `Bucket` + `Key` + `VersionId` + `TaskStatus` + `ErrorCode` + `HTTPStatusCode` + `ResultMessage` **Example — Lambda function task report** In the following example, the Lambda function successfully copied the Amazon S3 object to another bucket. The returned Amazon S3 response is passed back to S3 Batch Operations and is then written into the final completion report. ``` amzn-s3-demo-bucket1,image_17775,,succeeded,200,,"{u'CopySourceVersionId': 'xVR78haVKlRnurYofbTfYr3ufYbktF8h', u'CopyObjectResult': {u'LastModified': datetime.datetime(2019, 4, 5, 17, 35, 39, tzinfo=tzlocal()), u'ETag': '""fe66f4390c50f29798f040d7aae72784""'}, 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': 'nXNaClIMxEJzWNmeMNQV2KpjbaCJLn0OGoXWZpuVOFS/iQYWxb3QtTvzX9SVfx2lA3oTKLwImKw=', 'RequestId': '3ED5852152014362', 'HTTPHeaders': {'content-length': '234', 'x-amz-id-2': 'nXNaClIMxEJzWNmeMNQV2KpjbaCJLn0OGoXWZpuVOFS/iQYWxb3QtTvzX9SVfx2lA3oTKLwImKw=', 'x-amz-copy-source-version-id': 'xVR78haVKlRnurYofbTfYr3ufYbktF8h', 'server': 'AmazonS3', 'x-amz-request-id': '3ED5852152014362', 'date': 'Fri, 05 Apr 2019 17:35:39 GMT', 'content-type': 'application/xml'}}}" amzn-s3-demo-bucket1,image_17763,,succeeded,200,,"{u'CopySourceVersionId': '6HjOUSim4Wj6BTcbxToXW44pSZ.40pwq', u'CopyObjectResult': {u'LastModified': datetime.datetime(2019, 4, 5, 17, 35, 39, tzinfo=tzlocal()), u'ETag': '""fe66f4390c50f29798f040d7aae72784""'}, 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': 'GiCZNYr8LHd/Thyk6beTRP96IGZk2sYxujLe13TuuLpq6U2RD3we0YoluuIdm1PRvkMwnEW1aFc=', 'RequestId': '1BC9F5B1B95D7000', 'HTTPHeaders': {'content-length': '234', 'x-amz-id-2': 'GiCZNYr8LHd/Thyk6beTRP96IGZk2sYxujLe13TuuLpq6U2RD3we0YoluuIdm1PRvkMwnEW1aFc=', 'x-amz-copy-source-version-id': '6HjOUSim4Wj6BTcbxToXW44pSZ.40pwq', 'server': 'AmazonS3', 'x-amz-request-id': '1BC9F5B1B95D7000', 'date': 'Fri, 05 Apr 2019 17:35:39 GMT', 'content-type': 'application/xml'}}}" amzn-s3-demo-bucket1,image_17860,,succeeded,200,,"{u'CopySourceVersionId': 'm.MDD0g_QsUnYZ8TBzVFrp.TmjN8PJyX', u'CopyObjectResult': {u'LastModified': datetime.datetime(2019, 4, 5, 17, 35, 40, tzinfo=tzlocal()), u'ETag': '""fe66f4390c50f29798f040d7aae72784""'}, 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': 'F9ooZOgpE5g9sNgBZxjdiPHqB4+0DNWgj3qbsir+sKai4fv7rQEcF2fBN1VeeFc2WH45a9ygb2g=', 'RequestId': '8D9CA56A56813DF3', 'HTTPHeaders': {'content-length': '234', 'x-amz-id-2': 'F9ooZOgpE5g9sNgBZxjdiPHqB4+0DNWgj3qbsir+sKai4fv7rQEcF2fBN1VeeFc2WH45a9ygb2g=', 'x-amz-copy-source-version-id': 'm.MDD0g_QsUnYZ8TBzVFrp.TmjN8PJyX', 'server': 'AmazonS3', 'x-amz-request-id': '8D9CA56A56813DF3', 'date': 'Fri, 05 Apr 2019 17:35:40 GMT', 'content-type': 'application/xml'}}}" ``` The following example report shows a case in which the AWS Lambda function timed out, causing failures to exceed the failure threshold. It is then marked as a `PermanentFailure`. ``` amzn-s3-demo-bucket1,image_14975,,failed,200,PermanentFailure,"Lambda returned function error: {""errorMessage"":""2019-04-05T17:35:21.155Z 2845ca0d-38d9-4c4b-abcf-379dc749c452 Task timed out after 3.00 seconds""}" amzn-s3-demo-bucket1,image_15897,,failed,200,PermanentFailure,"Lambda returned function error: {""errorMessage"":""2019-04-05T17:35:29.610Z 2d0a330b-de9b-425f-b511-29232fde5fe4 Task timed out after 3.00 seconds""}" amzn-s3-demo-bucket1,image_14819,,failed,200,PermanentFailure,"Lambda returned function error: {""errorMessage"":""2019-04-05T17:35:22.362Z fcf5efde-74d4-4e6d-b37a-c7f18827f551 Task timed out after 3.00 seconds""}" amzn-s3-demo-bucket1,image_15930,,failed,200,PermanentFailure,"Lambda returned function error: {""errorMessage"":""2019-04-05T17:35:29.809Z 3dd5b57c-4a4a-48aa-8a35-cbf027b7957e Task timed out after 3.00 seconds""}" amzn-s3-demo-bucket1,image_17644,,failed,200,PermanentFailure,"Lambda returned function error: {""errorMessage"":""2019-04-05T17:35:46.025Z 10a764e4-2b26-4d8c-9056-1e1072b4723f Task timed out after 3.00 seconds""}" amzn-s3-demo-bucket1,image_17398,,failed,200,PermanentFailure,"Lambda returned function error: {""errorMessage"":""2019-04-05T17:35:44.661Z 1e306352-4c54-4eba-aee8-4d02f8c0235c Task timed out after 3.00 seconds""}" ``` **Example — Compute checksum task report** In the following example, the **Compute checksum** operation successfully calculated the checksum for the uploaded object while at rest. The returned Amazon S3 response is passed back to S3 Batch Operations, and is then written into the final completion report: ``` amzn-s3-demo-bucket1,s3-standard-1mb-test-object,,succeeded,200,,"{""checksum_base64"":""bS9TOQ\u003d\u003d"",""etag"":""3c3c1813042989094598e4b57ecbdc82"",""checksumAlgorithm"":""CRC32"",""checksumType"":""FULL_OBJECT"",""checksum_hex"":""6D2F5339""}" ``` The following example report shows what happens when a **Compute checksum** operation fails, resulting in a failed task report: ``` amzn-s3-demo-bucket1,image_14975,,failed,200,PermanentFailure,"error details: {""failureMessage"":"Task 2845ca0d-38d9-4c4b-abcf-379dc749c452 SSE-C encryption type is not supported for this operation", ""errorCode"": "400"}" amzn-s3-demo-bucket1,image_14975,,failed,200,PermanentFailure,"error details: {""failureMessage"":"Task 2845ca0d-38d9-4c4b-abcf-379dc749c452 Key not found", ""errorCode"": "404"}" amzn-s3-demo-bucket1,image_14975,,failed,200,PermanentFailure,"error details: {""failureMessage"":"Task 2845ca0d-38d9-4c4b-abcf-379dc749c452 Internal server error, please retry", ""errorCode"": "500"}" ``` # Controlling access and labeling jobs using tags You can label and control access to your S3 Batch Operations jobs by adding *tags*. Tags can be used to identify who is responsible for a Batch Operations job. The presence of job tags can grant or limit a user's ability to cancel a job, activate a job in the confirmation state, or change a job's priority level. You can create jobs with tags attached to them, and you can add tags to jobs after they're created. Each tag is a key-value pair that can be included when you create the job or updated later. **Warning** Make sure that your job tags don't contain any confidential information or personal data. Consider the following tagging example: Suppose that you want your Finance department to create a Batch Operations job. You could write an AWS Identity and Access Management (IAM) policy that allows a user to invoke `CreateJob`, provided that the job is created with the `Department` tag assigned the value `Finance`. Furthermore, you could attach that policy to all users who are members of the Finance department. Continuing with this example, you could write a policy that allows a user to update the priority of any job that has the desired tags, or cancel any job that has those tags. For more information, see [Controlling permissions for Batch Operations using job tags](batch-ops-job-tags-examples.md). You can add tags to new S3 Batch Operations jobs when you create them, or you can add them to existing jobs. Note the following tag restrictions: + You can associate up to 50 tags with a job as long as they have unique tag keys. + A tag key can be up to 128 Unicode characters in length, and tag values can be up to 256 Unicode characters in length. + The key and values are case sensitive. For more information about tag restrictions, see [User-Defined Tag Restrictions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html) in the *AWS Billing and Cost Management User Guide*. ## API operations related to S3 Batch Operations job tagging Amazon S3 supports the following API operations that are specific to S3 Batch Operations job tagging: + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_GetJobTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_GetJobTagging.html) – Returns the tag set associated with a Batch Operations job. + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_PutJobTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_PutJobTagging.html) – Replaces the tag set associated with a job. There are two distinct scenarios for S3 Batch Operations job tag management using this API action: + Job has no tags – You can add a set of tags to a job (the job has no prior tags). + Job has a set of existing tags – To modify the existing tag set, you can either replace the existing tag set entirely, or make changes within the existing tag set by retrieving the existing tag set using [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_GetJobTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_GetJobTagging.html), modify that tag set, and use this API action to replace the tag set with the one you have modified. **Note** If you send this request with an empty tag set, S3 Batch Operations deletes the existing tag set on the object. If you use this method, you are charged for a Tier 1 Request (`PUT`). For more information, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing). To delete existing tags for your Batch Operations job, the `DeleteJobTagging` action is preferred because it achieves the same result without incurring charges. + [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_DeleteJobTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_DeleteJobTagging.html) – Deletes the tag set associated with a Batch Operations job. # Creating a Batch Operations job with job tags used for labeling You can label and control access to your Amazon S3 Batch Operations jobs by adding *tags*. Tags can be used to identify who is responsible for a Batch Operations job. You can create jobs with tags attached to them, and you can add tags to jobs after they are created. For more information, see [Controlling access and labeling jobs using tags](batch-ops-job-tags.md). ## Using the AWS CLI The following AWS CLI example creates an S3 Batch Operations `S3PutObjectCopy` job using job tags as labels for the job. 1. Select the action or `OPERATION` that you want the Batch Operations job to perform, and choose your `TargetResource`. ``` read -d '' OPERATION < To create an S3 Batch Operations job with tags using the AWS SDK for Java, you can use the S3Control client to configure job parameters including manifest location, job operations, reporting settings, and tags for organization and tracking purposes. For examples of how to create S3 Batch Operations jobs with tags using the AWS SDK for Java, see [Create a batch job to copy objects](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_CreateJob_section.html) in the *Amazon S3 API Reference*. # Deleting the tags from an S3 Batch Operations job You can use these examples to delete the tags from an Amazon S3 Batch Operations job. ## Using the AWS CLI The following example deletes the tags from a Batch Operations job using the AWS CLI. ``` aws \ s3control delete-job-tagging \ --account-id 123456789012 \ --job-id Example-e25a-4ed2-8bee-7f8ed7fc2f1c \ --region us-east-1 ``` ## Delete the job tags of a Batch Operations job To delete the tags of an S3 Batch Operations job using the AWS SDK for Java, you can use the S3Control client with the job ID to remove all tags associated with the batch operations job. For examples of how to delete job tags with the AWS SDK for Java, see [Delete tags from a batch job](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_DeleteJobTagging_section.html) in the *Amazon S3 API Reference*. # Adding job tags to an existing Batch Operations job You can use the [https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_PutJobTagging.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_PutJobTagging.html) API operation to add job tags to your existing Amazon S3 Batch Operations jobs. For more information, see the following examples. ## Using the AWS CLI The following is an example of using `s3control put-job-tagging` to add job tags to your S3 Batch Operations job by using the AWS CLI. To use the examples, replace the *`user input placeholders`* with your own information. **Note** If you send this request with an empty tag set, Batch Operations deletes the existing tag set on the object. However, if you use this approach, you are charged for a Tier 1 Request (`PUT`). For more information, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing). Instead, to delete existing tags for your Batch Operations job, we recommend using the `DeleteJobTagging` operation because it achieves the same result without incurring charges. 1. Identify the job `TAGS` that you want for the job. In this case, you apply two tags, `department` and `FiscalYear`, with the values `Marketing` and `2020` respectively. ``` read -d '' TAGS < To put tags on an S3 Batch Operations job using the AWS SDK for Java, you can use the S3Control client to add or update tags with key-value pairs for organization and tracking purposes. For examples of how to put job tags with the AWS SDK for Java, see [Add tags to a batch job](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_PutJobTagging_section.html) in the *Amazon S3 API Reference*. # Getting the tags of a S3 Batch Operations job To retrieve the tags of an Amazon S3 Batch Operations job, you can use the `GetJobTagging` API operation. For more information, see the following examples. ## Using the AWS CLI The following example gets the tags of a Batch Operations job using the AWS CLI. To use this example, replace the *`user input placeholders`* with your own information. ``` aws \ s3control get-job-tagging \ --account-id 123456789012 \ --job-id Example-e25a-4ed2-8bee-7f8ed7fc2f1c \ --region us-east-1 ``` ## Using the AWS SDK for Java To get the tags of an S3 Batch Operations job using the AWS SDK for Java, you can use the S3Control client with the job ID to retrieve all tags associated with the batch operations job and return them as a list. For examples of how to get job tags with the AWS SDK for Java, see [Get tags from a batch job](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_GetJobTagging_section.html) in the *Amazon S3 API Reference*. # Controlling permissions for Batch Operations using job tags To help you manage your Amazon S3 Batch Operations jobs, you can add *job tags*. With job tags, you can control access to your Batch Operations jobs and enforce that tags be applied when any job is created. You can apply up to 50 job tags to each Batch Operations job. By using tags, you can set granular policies to restrict the set of users that can edit the job. Job tags can grant or limit a user’s ability to cancel a job, activate a job in the confirmation state, or change a job’s priority level. In addition, you can enforce that tags be applied to all new jobs, and specify the allowed key-value pairs for the tags. You can express all of these conditions by using [AWS Identity and Access Management (IAM) policy language](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_iam-tags.html). For more information, see [Actions, resources, and condition keys for Amazon S3](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html) in the *Service Authorization Reference*. For more information about the permissions to S3 API operations by S3 resource types, see [Required permissions for Amazon S3 API operations](using-with-s3-policy-actions.md). The following example shows how you can use S3 Batch Operations job tags to grant users permission to create and edit only the jobs that are run within a specific department (for example, the Finance or Compliance department). You can also assign jobs based on the stage of development that they are related to, such as QA or Production. In this example, you use S3 Batch Operations job tags in IAM policies to grant users permission to create and edit only the jobs that are being run within their department. You assign jobs based on the stage of development that they are related to, such as QA or Production. The following examples use the following departments, with each department using Batch Operations in different ways: + Finance + Compliance + Business Intelligence + Engineering **Topics** + [ ## Controlling access by assigning tags to users and resources ](#job-tags-examples-attaching-tags) + [ ## Tagging Batch Operations jobs by stage and enforcing limits on job priority ](#tagging-jobs-by-stage-and-enforcing-limits-on-job-priority) ## Controlling access by assigning tags to users and resources In this scenario, the administrators are using [attribute-based access control (ABAC)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_attribute-based-access-control.html). ABAC is an IAM authorization strategy that defines permissions by attaching tags to both users and AWS resources. Users and jobs are assigned one of the following department tags: **Key : Value** + `department : Finance` + `department : Compliance` + `department : BusinessIntelligence` + `department : Engineering` **Note** Job tag keys and values are case sensitive. Using the ABAC access control strategy, you grant a user in the Finance department permission to create and manage S3 Batch Operations jobs within their department by associating the tag `department=Finance` with their user. Furthermore, you can attach a managed policy to the IAM user that allows any user in their company to create or modify S3 Batch Operations jobs within their respective departments. The policy in this example includes three policy statements: + The first statement in the policy allows the user to create a Batch Operations job provided that the job creation request includes a job tag that matches their respective department. This is expressed using the `"${aws:PrincipalTag/department}"` syntax, which is replaced by the user’s department tag at policy evaluation time. The condition is satisfied when the value provided for the department tag in the request `("aws:RequestTag/department")` matches the user’s department. + The second statement in the policy allows users to change the priority of jobs or update a job’s status provided that the job the user is updating matches the user’s department. + The third statement allows a user to update a Batch Operations job’s tags at any time via a `PutJobTagging` request as long as (1) their department tag is preserved and (2) the job they’re updating is within their department. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:CreateJob", "Resource": "*", "Condition": { "StringEquals": { "aws:RequestTag/department": "${aws:PrincipalTag/department}" } } }, { "Effect": "Allow", "Action": [ "s3:UpdateJobPriority", "s3:UpdateJobStatus" ], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/department": "${aws:PrincipalTag/department}" } } }, { "Effect": "Allow", "Action": "s3:PutJobTagging", "Resource": "*", "Condition": { "StringEquals": { "aws:RequestTag/department": "${aws:PrincipalTag/department}", "aws:ResourceTag/department": "${aws:PrincipalTag/department}" } } } ] } ``` ------ ## Tagging Batch Operations jobs by stage and enforcing limits on job priority All S3 Batch Operations jobs have a numeric priority, which Amazon S3 uses to decide in what order to run the jobs. For this example, you restrict the maximum priority that most users can assign to jobs, with higher priority ranges reserved for a limited set of privileged users, as follows: + QA stage priority range (low): 1-100 + Production stage priority range (high): 1-300 To do this, introduce a new tag set representing the stage of the job: **Key : Value** + `stage : QA` + `stage : Production` ### Creating and updating low-priority jobs within a department This policy introduces two new restrictions on S3 Batch Operations job creation and update, in addition to the department-based restriction: + It allows users to create or update jobs in their department with a new condition that requires the job to include the tag `stage=QA`. + It allows users to create or update a job’s priority up to a new maximum priority of 100. ``` {         "Version": "2012-10-17",         "Statement": [         {         "Effect": "Allow",         "Action": "s3:CreateJob",         "Resource": "*",         "Condition": {             "StringEquals": {                 "aws:RequestTag/department": "${aws:PrincipalTag/department}",                 "aws:RequestTag/stage": "QA"             },             "NumericLessThanEquals": {                 "s3:RequestJobPriority": 100             }         }     },     {         "Effect": "Allow",         "Action": [             "s3:UpdateJobStatus"         ],         "Resource": "*",         "Condition": {             "StringEquals": {                 "aws:ResourceTag/department": "${aws:PrincipalTag/department}"             }         }     },     {         "Effect": "Allow",         "Action": "s3:UpdateJobPriority",         "Resource": "*",         "Condition": {             "StringEquals": {                 "aws:ResourceTag/department": "${aws:PrincipalTag/department}",                 "aws:ResourceTag/stage": "QA"             },             "NumericLessThanEquals": {                 "s3:RequestJobPriority": 100             }         }     },     {         "Effect": "Allow",         "Action": "s3:PutJobTagging",         "Resource": "*",         "Condition": {             "StringEquals": {                 "aws:RequestTag/department" : "${aws:PrincipalTag/department}",                 "aws:ResourceTag/department": "${aws:PrincipalTag/department}",                 "aws:RequestTag/stage": "QA",                 "aws:ResourceTag/stage": "QA"             }         }     },     {         "Effect": "Allow",         "Action": "s3:GetJobTagging",         "Resource": "*"     }     ] } ``` ### Creating and updating high-priority jobs within a department A small number of users might require the ability to create high priority jobs in either QA or Production. To support this need, you create a managed policy that’s adapted from the low-priority policy in the previous section. This policy does the following: + Allows users to create or update jobs in their department with either the tag `stage=QA` or `stage=Production`. + Allows users to create or update a job’s priority up to a maximum of 300. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:CreateJob", "Resource": "*", "Condition": { "ForAnyValue:StringEquals": { "aws:RequestTag/stage": [ "QA", "Production" ] }, "StringEquals": { "aws:RequestTag/department": "${aws:PrincipalTag/department}" }, "NumericLessThanEquals": { "s3:RequestJobPriority": 300 } } }, { "Effect": "Allow", "Action": [ "s3:UpdateJobStatus" ], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/department": "${aws:PrincipalTag/department}" } } }, { "Effect": "Allow", "Action": "s3:UpdateJobPriority", "Resource": "*", "Condition": { "ForAnyValue:StringEquals": { "aws:ResourceTag/stage": [ "QA", "Production" ] }, "StringEquals": { "aws:ResourceTag/department": "${aws:PrincipalTag/department}" }, "NumericLessThanEquals": { "s3:RequestJobPriority": 300 } } }, { "Effect": "Allow", "Action": "s3:PutJobTagging", "Resource": "*", "Condition": { "StringEquals": { "aws:RequestTag/department": "${aws:PrincipalTag/department}", "aws:ResourceTag/department": "${aws:PrincipalTag/department}" }, "ForAnyValue:StringEquals": { "aws:RequestTag/stage": [ "QA", "Production" ], "aws:ResourceTag/stage": [ "QA", "Production" ] } } } ] } ``` ------ # Managing S3 Object Lock using S3 Batch Operations You can use S3 Batch Operations to perform large-scale batch operations on Amazon S3 objects. S3 Batch Operations can perform a single operation on lists of Amazon S3 objects that you specify. A single job can perform a specified operation on billions of objects containing exabytes of data. Amazon S3 tracks progress, sends notifications, and stores a detailed completion report of all actions, providing a fully managed, auditable, and serverless experience. You can use S3 Batch Operations through the Amazon S3 console, AWS CLI, AWS SDKs, or Amazon S3 REST API. With S3 Object Lock, you can place a legal hold on an object version. Like setting a retention period, a legal hold prevents an object version from being overwritten or deleted. However, a legal hold doesn't have an associated retention period and remains in effect until the legal hold is removed. For more information, see [S3 Object Lock legal hold](batch-ops-legal-hold.md). To use S3 Batch Operations with Object Lock to add legal holds to many Amazon S3 objects at once, see the following topics. **Topics** + [ # Enabling S3 Object Lock using S3 Batch Operations ](batch-ops-object-lock.md) + [ # Setting Object Lock retention using Batch Operations ](batch-ops-object-lock-retention.md) + [ # Using S3 Batch Operations with S3 Object Lock retention compliance mode ](batch-ops-compliance-mode.md) + [ # Use S3 Batch Operations with S3 Object Lock retention governance mode ](batch-ops-governance-mode.md) + [ # Using S3 Batch Operations to turn off S3 Object Lock legal holds ](batch-ops-legal-hold-off.md) # Enabling S3 Object Lock using S3 Batch Operations You can use Amazon S3 Batch Operations with S3 Object Lock to manage retention or enable a legal hold for many Amazon S3 objects at once. You specify the list of target objects in your manifest and submit it to Batch Operations for completion. For more information, see [S3 Object Lock retention](batch-ops-retention-date.md) and [S3 Object Lock legal hold](batch-ops-legal-hold.md). The following examples show how to create an AWS Identity and Access Management (IAM) role with S3 Batch Operations permissions and update the role permissions to create jobs that enable Object Lock. You must also have a `CSV` manifest that identifies the objects for your S3 Batch Operations job. For more information, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest). To use the following examples, replace the *`user input placeholders`* with your own information. ## Using the AWS CLI 1. Create an IAM role and assign S3 Batch Operations permissions to run. This step is required for all S3 Batch Operations jobs. ``` export AWS_PROFILE='aws-user' read -d '' batch_operations_trust_policy < You can create an IAM role with S3 Batch Operations permissions, and update the role permissions to create jobs that enable Object Lock by using the AWS SDK for Java. You must also have a `CSV` manifest identifying the objects for your S3 Batch Operations job. For more information, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest). Perform the following steps: 1. Create an IAM role and assign S3 Batch Operations permissions to run. This step is required for all S3 Batch Operations jobs. 1. Set up S3 Batch Operations with S3 Object Lock to run. You allow the role to do the following: 1. Run Object Lock on the S3 bucket that contains the target objects that you want Batch Operations to run on. 1. Read the S3 bucket where the manifest CSV file and the objects are located. 1. Write the results of the S3 Batch Operations job to the reporting bucket. For a code example that demonstrates how to create an IAM role for enabling S3 Object Lock using S3 Batch Operations with the AWS SDK for Java, see [CreateObjectLockRole.java](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/s3/src/main/java/com/example/s3/batch/CreateObjectLockRole.java) in the *AWS SDK for Java 2.x Code Examples*. # Setting Object Lock retention using Batch Operations You can use Amazon S3 Batch Operations with S3 Object Lock to manage retention for many Amazon S3 objects at once. You specify the list of target objects in your manifest and submit it to Batch Operations for completion. For more information, see [S3 Object Lock retention](batch-ops-retention-date.md) and [S3 Object Lock legal hold](batch-ops-legal-hold.md). The following examples show how to create an AWS Identity and Access Management (IAM) role with S3 Batch Operations permissions and update the role permissions to include the `s3:PutObjectRetention` permissions so that you can run S3 Object Lock retention on the objects in your manifest bucket. You must also have a `CSV` manifest that identifies the objects for your S3 Batch Operations job. For more information, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest). To use the following examples, replace the *`user input placeholders`* with your own information. ## Using the AWS CLI The following AWS CLI example shows how to use Batch Operations to apply S3 Object Lock retention across multiple objects. ``` export AWS_PROFILE='aws-user' read -d '' retention_permissions < For examples of how to use Batch Operations to apply S3 Object Lock retention across multiple objects with the AWS SDK for Java, see [Use CreateJob with an AWS SDK or CLI](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_CreateJob_section.html) in the *Amazon S3 API Reference*. # Using S3 Batch Operations with S3 Object Lock retention compliance mode The following example builds on the previous examples of creating a trust policy and setting S3 Batch Operations and S3 Object Lock configuration permissions on your objects. This example sets the retention mode to `COMPLIANCE` and the `retain until date` to January 1, 2025. This example creates a job that targets objects in the manifest bucket and reports the results in the reports bucket that you identified. To use the following examples, replace the *`user input placeholders`* with your own information. ## Using the AWS CLI The following AWS CLI examples show how to use Batch Operations to apply S3 Object Lock retention compliance mode across multiple objects. **Example — Set S3 Object Lock retention compliance mode across multiple objects** ``` export AWS_PROFILE='aws-user' export AWS_DEFAULT_REGION='us-west-2' export ACCOUNT_ID=123456789012 export ROLE_ARN='arn:aws:iam::123456789012:role/batch_operations-objectlock' read -d '' OPERATION < The following AWS SDK for Java examples show how to use Batch Operations to apply S3 Object Lock retention compliance mode across multiple objects, including setting the retention mode to COMPLIANCE with a retain until date and extending the COMPLIANCE mode retain until date. For examples of how to use Batch Operations to apply S3 Object Lock retention compliance mode across multiple objects with the AWS SDK for Java, see [Use CreateJob with an AWS SDK or CLI](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_CreateJob_section.html) in the *Amazon S3 API Reference*. # Use S3 Batch Operations with S3 Object Lock retention governance mode The following example builds on the previous example of creating a trust policy, and setting S3 Batch Operations and S3 Object Lock configuration permissions. This example shows how to apply S3 Object Lock retention governance with the `retain until date` of January 30, 2025, across multiple objects. It creates a Batch Operations job that uses the manifest bucket and reports the results in the reports bucket. To use the following examples, replace the *`user input placeholders`* with your own information. ## Using the AWS CLI The following AWS CLI examples show how to use Batch Operations to apply S3 Object Lock retention governance mode across multiple objects. **Example — Apply S3 Object Lock retention governance across multiple objects with the retain until date of January 30, 2025** ``` export AWS_PROFILE='aws-user' export AWS_DEFAULT_REGION='us-west-2' export ACCOUNT_ID=123456789012 export ROLE_ARN='arn:aws:iam::123456789012:role/batch_operations-objectlock' read -d '' OPERATION < The following AWS SDK for Java examples show how to apply S3 Object Lock retention governance with the `retain until date` set to January 30, 2025 across multiple objects, including applying Object Lock retention governance across multiple objects with a retain until date and bypassing retention governance across multiple objects. For examples of how to use Batch Operations with S3 Object Lock retention governance mode with the AWS SDK for Java, see [Use CreateJob with an AWS SDK or CLI](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_CreateJob_section.html) in the *Amazon S3 API Reference*. # Using S3 Batch Operations to turn off S3 Object Lock legal holds The following example builds on the previous examples of creating a trust policy, and setting S3 Batch Operations and S3 Object Lock configuration permissions. This example shows how to disable Object Lock legal hold on objects using Batch Operations. The example first updates the role to grant `s3:PutObjectLegalHold` permissions, creates a Batch Operations job that turns off (removes) legal hold from the objects identified in the manifest, and then reports on it. To use the following examples, replace the *`user input placeholders`* with your own information. ## Using the AWS CLI The following AWS CLI examples show how to use Batch Operations to turn off S3 Object Lock legal holds across multiple objects. **Example — Updates the role to grant `s3:PutObjectLegalHold` permissions** ``` export AWS_PROFILE='aws-user' read -d '' legal_hold_permissions < For examples of how to use S3 Batch Operations to turn off S3 Object Lock legal hold with the AWS SDK for Java, see [Use CreateJob with an AWS SDK or CLI](https://docs.aws.amazon.com/AmazonS3/latest/API/s3-control_example_s3-control_CreateJob_section.html) in the *Amazon S3 API Reference*. # Tutorial: Batch-transcoding videos with S3 Batch Operations Video consumers use devices of all shapes, sizes, and vintages to enjoy media content. This wide array of devices presents a challenge for content creators and distributors. Instead of being in a one-size-fits-all format, videos must be converted so that they can span a broad range of sizes, formats, and bitrates. This conversion task is even more challenging when you have a large number of videos that must be converted. AWS offers you a method to build a scalable, distributed architecture that does the following: + Ingests input videos + Processes the videos for playback on a wide range of devices + Stores the transcoded media files + Delivers the output media files to meet demand When you have extensive video repositories stored in Amazon S3, you can transcode these videos from their source formats into multiple file types in the size, resolution, and format needed by a particular video player or device. Specifically, [S3 Batch Operations](https://aws.amazon.com/s3/features/batch-operations) provides you with a solution to invoke AWS Lambda functions for existing input videos in an S3 source bucket. Then, the Lambda functions call [AWS Elemental MediaConvert](https://aws.amazon.com/mediaconvert/) to perform large-scale video transcoding tasks. The converted output media files are stored in an S3 destination bucket. ![\[A batch-transcoding workflow diagram.\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/images/batchops-example-image-global.png) **Objective** In this tutorial, you learn how to set up S3 Batch Operations to invoke a Lambda function for batch-transcoding of videos stored in an S3 source bucket. The Lambda function calls MediaConvert to transcode the videos. The outputs for each video in the S3 source bucket are as follows: + An [HTTP Live Streaming (HLS)](http://wikipedia.org/wiki/HTTP_Live_Streaming) adaptive bitrate stream for playback on devices of multiple sizes and varying bandwidths + An MP4 video file + Thumbnail images collected at intervals **Topics** + [ ## Prerequisites ](#batchops-s3-prerequisites) + [ ## Step 1: Create an S3 bucket for the output media files ](#batchops-s3-step1) + [ ## Step 2: Create an IAM role for MediaConvert ](#batchops-s3-step2) + [ ## Step 3: Create an IAM role for your Lambda function ](#batchops-s3-step3) + [ ## Step 4: Create a Lambda function for video transcoding ](#batchops-s3-step4) + [ ## Step 5: Configure Amazon S3 Inventory for your S3 source bucket ](#batchops-s3-step5) + [ ## Step 6: Create an IAM role for S3 Batch Operations ](#batchops-s3-step6) + [ ## Step 7: Create and run an S3 Batch Operations job ](#batchops-s3-step7) + [ ## Step 8: Check the output media files from your S3 destination bucket ](#batchops-s3-step8) + [ ## Step 9: Clean up ](#batchops-s3-step9) + [ ## Next steps ](#batchops-s3-next-steps) ## Prerequisites Before you start this tutorial, you must have an Amazon S3 source bucket (for example, `amzn-s3-demo-source-bucket`) with videos to be transcoded already stored in it. You can give the bucket another name if you want. For more information about bucket names in Amazon S3, see [General purpose bucket naming rules](bucketnamingrules.md). For the S3 source bucket, keep the settings related to **Block Public Access settings for this bucket** set to the defaults (**Block *all* public access** is enabled). For more information, see [Creating a general purpose bucket](create-bucket-overview.md). For more information about uploading videos to the S3 source bucket, see [Uploading objects](upload-objects.md). If you're uploading many large video files to S3, you might want to use [Amazon S3 Transfer Acceleration](https://aws.amazon.com/s3/transfer-acceleration) to configure fast and secure file transfers. Transfer Acceleration can speed up video uploading to your S3 bucket for long-distance transfer of larger videos. For more information, see [Configuring fast, secure file transfers using Amazon S3 Transfer Acceleration](transfer-acceleration.md). ## Step 1: Create an S3 bucket for the output media files In this step, you create an S3 destination bucket to store the converted output media files. You also create a Cross Origin Resource Sharing (CORS) configuration to allow cross-origin access to the transcoded media files stored in your S3 destination bucket. **Topics** + [ ### Create a bucket for the output media files ](#batchops-s3-step1-create-bucket) + [ ### Add a CORS configuration to the S3 output bucket ](#batchops-s3-step1-cors) ### Create a bucket for the output media files 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. Choose **Create bucket**. 1. For **Bucket name**, enter a name for your bucket (for example, `amzn-s3-demo-destination-bucket1`). 1. For **Region**, choose the AWS Region where you want the bucket to reside. 1. To ensure public access to your output media files, in **Block Public Access settings for this bucket**, clear **Block *all* public access**. **Warning** Before you complete this step, review [Blocking public access to your Amazon S3 storage](access-control-block-public-access.md) to ensure that you understand and accept the risks involved with allowing public access. When you turn off Block Public Access settings to make your bucket public, anyone on the internet can access your bucket. We recommend that you block all public access to your buckets. If you don’t want to clear the Block Public Access settings, you can use Amazon CloudFront to deliver the transcoded media files to viewers (end users). For more information, see [Tutorial: Hosting on-demand streaming video with Amazon S3, Amazon CloudFront, and Amazon Route 53](tutorial-s3-cloudfront-route53-video-streaming.md). 1. Select the check box next to **I acknowledge that the current settings might result in this bucket and the objects within becoming public.** 1. Keep the remaining settings set to the defaults. 1. Choose **Create bucket**. ### Add a CORS configuration to the S3 output bucket A JSON CORS configuration defines a way for client web applications (video players in this context) that are loaded in one domain to play transcoded output media files in a different domain. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the name of the bucket that you created earlier (for example, `amzn-s3-demo-destination-bucket1`). 1. Choose the **Permissions** tab. 1. In the **Cross-origin resource sharing (CORS)** section, choose **Edit**. 1. In the CORS configuration text box, copy and paste the following CORS configuration. The CORS configuration must be in JSON format. In this example, the `AllowedOrigins` attribute uses the wildcard character (`*`) to specify all origins. If you know your specific origin, you can restrict the `AllowedOrigins` attribute to your specific player URL. For more information about configuring this and other attributes, see [Elements of a CORS configuration](ManageCorsUsing.md). ``` [ { "AllowedOrigins": [ "*" ], "AllowedMethods": [ "GET" ], "AllowedHeaders": [ "*" ], "ExposeHeaders": [] } ] ``` 1. Choose **Save changes**. ## Step 2: Create an IAM role for MediaConvert To use AWS Elemental MediaConvert to transcode input videos stored in your S3 bucket, you must have an AWS Identity and Access Management (IAM) service role to grant MediaConvert permissions to read and write video files from and to your S3 source and destination buckets. When you run transcoding jobs, the MediaConvert console uses this role. **To create an IAM role for MediaConvert** 1. Create an IAM role with a role name that you choose (for example, **tutorial-mediaconvert-role**). To create this role, follow the steps in [Create your MediaConvert role in IAM (console)](https://docs.aws.amazon.com/mediaconvert/latest/ug/creating-the-iam-role-in-iam.html) in the *AWS Elemental MediaConvert User Guide*. 1. After you create the IAM role for MediaConvert, in the list of **Roles**, choose the name of the role for MediaConvert that you created (for example, **tutorial-mediaconvert-role**). 1. On the **Summary** page, copy the **Role ARN** (which starts with `arn:aws:iam::`), and save the ARN for use later. For more information about ARNs, see [Amazon Resource Names (ARNs)](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) in the *AWS General Reference*. ## Step 3: Create an IAM role for your Lambda function To batch-transcode videos with MediaConvert and S3 Batch Operations, you use a Lambda function to connect these two services to convert videos. This Lambda function must have an IAM role that grants the Lambda function permissions to access MediaConvert and S3 Batch Operations. **Topics** + [ ### Create an IAM role for your Lambda function ](#batchops-s3-step3-role) + [ ### Embed an inline policy for the IAM role of your Lambda function ](#batchops-s3-step3-inline-policy) ### Create an IAM role for your Lambda function 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**, and then choose **Create role**. 1. Choose the **AWS service** role type, and then under **Common use cases**, choose **Lambda**. 1. Choose **Next: Permissions**. 1. On the **Attach permissions policies** page, enter **AWSLambdaBasicExecutionRole** in the **Filter policies** box. To attach the managed policy **AWSLambdaBasicExecutionRole** to this role to grant write permissions to Amazon CloudWatch Logs, select the check box next to **AWSLambdaBasicExecutionRole**. 1. Choose **Next**. 1. For **Role name**, enter **tutorial-lambda-transcode-role**. 1. (Optional) Add tags to the managed policy. 1. Choose **Create role**. ### Embed an inline policy for the IAM role of your Lambda function To grant permissions to the MediaConvert resource that's needed for the Lambda function to execute, you must use an inline policy. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**. 1. In the **Roles** list, choose the name of the IAM role that you created earlier for your Lambda function (for example, **tutorial-lambda-transcode-role**). 1. Choose the **Permissions** tab. 1. Choose **Add inline policy**. 1. Choose the **JSON** tab, and then copy and paste the following JSON policy. In the JSON policy, replace the example ARN value of `Resource` with the role ARN of the IAM role for MediaConvert that you created in [Step 2](#batchops-s3-step2) (for example, **tutorial-mediaconvert-role**). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "*", "Effect": "Allow", "Sid": "Logging" }, { "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::111122223333:role/tutorial-mediaconvert-role" ], "Effect": "Allow", "Sid": "PassRole" }, { "Action": [ "mediaconvert:*" ], "Resource": [ "*" ], "Effect": "Allow", "Sid": "MediaConvertService" }, { "Action": [ "s3:*" ], "Resource": [ "*" ], "Effect": "Allow", "Sid": "S3Service" } ] } ``` ------ 1. Choose **Review Policy**. 1. For **Name**, enter **tutorial-lambda-policy**. 1. Choose **Create Policy**. After you create an inline policy, it is automatically embedded in the IAM role of your Lambda function. ## Step 4: Create a Lambda function for video transcoding In this section of the tutorial, you build a Lambda function using the SDK for Python to integrate with S3 Batch Operations and MediaConvert. To start transcoding the videos already stored in your S3 source bucket, you run an S3 Batch Operations job that directly invokes the Lambda function for each video in the S3 source bucket. Then, the Lambda function submits a transcoding job for each video to MediaConvert. **Topics** + [ ### Write Lambda function code and create a deployment package ](#batchops-s3-step4-write-function) + [ ### Create a Lambda function with an execution role (console) ](#batchops-s3-step4-create-function) + [ ### Deploy your Lambda function with .zip file archives and configure the Lambda function (console) ](#batchops-s3-step4-deploy-function) ### Write Lambda function code and create a deployment package 1. On your local machine, create a folder named `batch-transcode`. 1. In the `batch-transcode` folder, create a file with JSON job settings. For example, you can use the settings provided in this section, and name the file `job.json`. A `job.json` file specifies the following: + Which files to transcode + How you want to transcode your input videos + What output media files you want to create + What to name the transcoded files + Where to save the transcoded files + Which advanced features to apply, and so on In this tutorial, we use the following `job.json` file to create the following outputs for each video in the S3 source bucket: + An HTTP Live Streaming (HLS) adaptive bitrate stream for playback on multiple devices of differing sizes and varying bandwidths + An MP4 video file + Thumbnail images collected at intervals This example `job.json` file uses Quality-Defined Variable Bitrate (QVBR) to optimize video quality. The HLS output is Apple-compliant (audio unmixed from video, segment duration of 6 seconds, and optimized video quality through auto QVBR). If you don't want to use the example settings provided here, you can generate a `job.json` specification based on your use case. To ensure consistency across your outputs, make sure that your input files have similar video and audio configurations. For any input files with different video and audio configurations, create separate automations (unique `job.json` settings). For more information, see [Example AWS Elemental MediaConvert job settings in JSON](https://docs.aws.amazon.com/mediaconvert/latest/ug/example-job-settings.html) in the *AWS Elemental MediaConvert User Guide*. ``` { "OutputGroups": [ { "CustomName": "HLS", "Name": "Apple HLS", "Outputs": [ { "ContainerSettings": { "Container": "M3U8", "M3u8Settings": { "AudioFramesPerPes": 4, "PcrControl": "PCR_EVERY_PES_PACKET", "PmtPid": 480, "PrivateMetadataPid": 503, "ProgramNumber": 1, "PatInterval": 0, "PmtInterval": 0, "TimedMetadata": "NONE", "VideoPid": 481, "AudioPids": [ 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492 ] } }, "VideoDescription": { "Width": 640, "ScalingBehavior": "DEFAULT", "Height": 360, "TimecodeInsertion": "DISABLED", "AntiAlias": "ENABLED", "Sharpness": 50, "CodecSettings": { "Codec": "H_264", "H264Settings": { "InterlaceMode": "PROGRESSIVE", "NumberReferenceFrames": 3, "Syntax": "DEFAULT", "Softness": 0, "GopClosedCadence": 1, "GopSize": 2, "Slices": 1, "GopBReference": "DISABLED", "MaxBitrate": 1200000, "SlowPal": "DISABLED", "SpatialAdaptiveQuantization": "ENABLED", "TemporalAdaptiveQuantization": "ENABLED", "FlickerAdaptiveQuantization": "DISABLED", "EntropyEncoding": "CABAC", "FramerateControl": "INITIALIZE_FROM_SOURCE", "RateControlMode": "QVBR", "CodecProfile": "MAIN", "Telecine": "NONE", "MinIInterval": 0, "AdaptiveQuantization": "HIGH", "CodecLevel": "AUTO", "FieldEncoding": "PAFF", "SceneChangeDetect": "TRANSITION_DETECTION", "QualityTuningLevel": "SINGLE_PASS_HQ", "FramerateConversionAlgorithm": "DUPLICATE_DROP", "UnregisteredSeiTimecode": "DISABLED", "GopSizeUnits": "SECONDS", "ParControl": "INITIALIZE_FROM_SOURCE", "NumberBFramesBetweenReferenceFrames": 2, "RepeatPps": "DISABLED" } }, "AfdSignaling": "NONE", "DropFrameTimecode": "ENABLED", "RespondToAfd": "NONE", "ColorMetadata": "INSERT" }, "OutputSettings": { "HlsSettings": { "AudioGroupId": "program_audio", "AudioRenditionSets": "program_audio", "SegmentModifier": "$dt$", "IFrameOnlyManifest": "EXCLUDE" } }, "NameModifier": "_360" }, { "ContainerSettings": { "Container": "M3U8", "M3u8Settings": { "AudioFramesPerPes": 4, "PcrControl": "PCR_EVERY_PES_PACKET", "PmtPid": 480, "PrivateMetadataPid": 503, "ProgramNumber": 1, "PatInterval": 0, "PmtInterval": 0, "TimedMetadata": "NONE", "TimedMetadataPid": 502, "VideoPid": 481, "AudioPids": [ 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492 ] } }, "VideoDescription": { "Width": 960, "ScalingBehavior": "DEFAULT", "Height": 540, "TimecodeInsertion": "DISABLED", "AntiAlias": "ENABLED", "Sharpness": 50, "CodecSettings": { "Codec": "H_264", "H264Settings": { "InterlaceMode": "PROGRESSIVE", "NumberReferenceFrames": 3, "Syntax": "DEFAULT", "Softness": 0, "GopClosedCadence": 1, "GopSize": 2, "Slices": 1, "GopBReference": "DISABLED", "MaxBitrate": 3500000, "SlowPal": "DISABLED", "SpatialAdaptiveQuantization": "ENABLED", "TemporalAdaptiveQuantization": "ENABLED", "FlickerAdaptiveQuantization": "DISABLED", "EntropyEncoding": "CABAC", "FramerateControl": "INITIALIZE_FROM_SOURCE", "RateControlMode": "QVBR", "CodecProfile": "MAIN", "Telecine": "NONE", "MinIInterval": 0, "AdaptiveQuantization": "HIGH", "CodecLevel": "AUTO", "FieldEncoding": "PAFF", "SceneChangeDetect": "TRANSITION_DETECTION", "QualityTuningLevel": "SINGLE_PASS_HQ", "FramerateConversionAlgorithm": "DUPLICATE_DROP", "UnregisteredSeiTimecode": "DISABLED", "GopSizeUnits": "SECONDS", "ParControl": "INITIALIZE_FROM_SOURCE", "NumberBFramesBetweenReferenceFrames": 2, "RepeatPps": "DISABLED" } }, "AfdSignaling": "NONE", "DropFrameTimecode": "ENABLED", "RespondToAfd": "NONE", "ColorMetadata": "INSERT" }, "OutputSettings": { "HlsSettings": { "AudioGroupId": "program_audio", "AudioRenditionSets": "program_audio", "SegmentModifier": "$dt$", "IFrameOnlyManifest": "EXCLUDE" } }, "NameModifier": "_540" }, { "ContainerSettings": { "Container": "M3U8", "M3u8Settings": { "AudioFramesPerPes": 4, "PcrControl": "PCR_EVERY_PES_PACKET", "PmtPid": 480, "PrivateMetadataPid": 503, "ProgramNumber": 1, "PatInterval": 0, "PmtInterval": 0, "TimedMetadata": "NONE", "VideoPid": 481, "AudioPids": [ 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492 ] } }, "VideoDescription": { "Width": 1280, "ScalingBehavior": "DEFAULT", "Height": 720, "TimecodeInsertion": "DISABLED", "AntiAlias": "ENABLED", "Sharpness": 50, "CodecSettings": { "Codec": "H_264", "H264Settings": { "InterlaceMode": "PROGRESSIVE", "NumberReferenceFrames": 3, "Syntax": "DEFAULT", "Softness": 0, "GopClosedCadence": 1, "GopSize": 2, "Slices": 1, "GopBReference": "DISABLED", "MaxBitrate": 5000000, "SlowPal": "DISABLED", "SpatialAdaptiveQuantization": "ENABLED", "TemporalAdaptiveQuantization": "ENABLED", "FlickerAdaptiveQuantization": "DISABLED", "EntropyEncoding": "CABAC", "FramerateControl": "INITIALIZE_FROM_SOURCE", "RateControlMode": "QVBR", "CodecProfile": "MAIN", "Telecine": "NONE", "MinIInterval": 0, "AdaptiveQuantization": "HIGH", "CodecLevel": "AUTO", "FieldEncoding": "PAFF", "SceneChangeDetect": "TRANSITION_DETECTION", "QualityTuningLevel": "SINGLE_PASS_HQ", "FramerateConversionAlgorithm": "DUPLICATE_DROP", "UnregisteredSeiTimecode": "DISABLED", "GopSizeUnits": "SECONDS", "ParControl": "INITIALIZE_FROM_SOURCE", "NumberBFramesBetweenReferenceFrames": 2, "RepeatPps": "DISABLED" } }, "AfdSignaling": "NONE", "DropFrameTimecode": "ENABLED", "RespondToAfd": "NONE", "ColorMetadata": "INSERT" }, "OutputSettings": { "HlsSettings": { "AudioGroupId": "program_audio", "AudioRenditionSets": "program_audio", "SegmentModifier": "$dt$", "IFrameOnlyManifest": "EXCLUDE" } }, "NameModifier": "_720" }, { "ContainerSettings": { "Container": "M3U8", "M3u8Settings": {} }, "AudioDescriptions": [ { "AudioSourceName": "Audio Selector 1", "CodecSettings": { "Codec": "AAC", "AacSettings": { "Bitrate": 96000, "CodingMode": "CODING_MODE_2_0", "SampleRate": 48000 } } } ], "OutputSettings": { "HlsSettings": { "AudioGroupId": "program_audio", "AudioTrackType": "ALTERNATE_AUDIO_AUTO_SELECT_DEFAULT" } }, "NameModifier": "_audio" } ], "OutputGroupSettings": { "Type": "HLS_GROUP_SETTINGS", "HlsGroupSettings": { "ManifestDurationFormat": "INTEGER", "SegmentLength": 6, "TimedMetadataId3Period": 10, "CaptionLanguageSetting": "OMIT", "Destination": "s3://EXAMPLE-BUCKET/HLS/", "DestinationSettings": { "S3Settings": { "AccessControl": { "CannedAcl": "PUBLIC_READ" } } }, "TimedMetadataId3Frame": "PRIV", "CodecSpecification": "RFC_4281", "OutputSelection": "MANIFESTS_AND_SEGMENTS", "ProgramDateTimePeriod": 600, "MinSegmentLength": 0, "DirectoryStructure": "SINGLE_DIRECTORY", "ProgramDateTime": "EXCLUDE", "SegmentControl": "SEGMENTED_FILES", "ManifestCompression": "NONE", "ClientCache": "ENABLED", "StreamInfResolution": "INCLUDE" } } }, { "CustomName": "MP4", "Name": "File Group", "Outputs": [ { "ContainerSettings": { "Container": "MP4", "Mp4Settings": { "CslgAtom": "INCLUDE", "FreeSpaceBox": "EXCLUDE", "MoovPlacement": "PROGRESSIVE_DOWNLOAD" } }, "VideoDescription": { "Width": 1280, "ScalingBehavior": "DEFAULT", "Height": 720, "TimecodeInsertion": "DISABLED", "AntiAlias": "ENABLED", "Sharpness": 100, "CodecSettings": { "Codec": "H_264", "H264Settings": { "InterlaceMode": "PROGRESSIVE", "ParNumerator": 1, "NumberReferenceFrames": 3, "Syntax": "DEFAULT", "Softness": 0, "GopClosedCadence": 1, "HrdBufferInitialFillPercentage": 90, "GopSize": 2, "Slices": 2, "GopBReference": "ENABLED", "HrdBufferSize": 10000000, "MaxBitrate": 5000000, "ParDenominator": 1, "EntropyEncoding": "CABAC", "RateControlMode": "QVBR", "CodecProfile": "HIGH", "MinIInterval": 0, "AdaptiveQuantization": "AUTO", "CodecLevel": "AUTO", "FieldEncoding": "PAFF", "SceneChangeDetect": "ENABLED", "QualityTuningLevel": "SINGLE_PASS_HQ", "UnregisteredSeiTimecode": "DISABLED", "GopSizeUnits": "SECONDS", "ParControl": "SPECIFIED", "NumberBFramesBetweenReferenceFrames": 3, "RepeatPps": "DISABLED", "DynamicSubGop": "ADAPTIVE" } }, "AfdSignaling": "NONE", "DropFrameTimecode": "ENABLED", "RespondToAfd": "NONE", "ColorMetadata": "INSERT" }, "AudioDescriptions": [ { "AudioTypeControl": "FOLLOW_INPUT", "AudioSourceName": "Audio Selector 1", "CodecSettings": { "Codec": "AAC", "AacSettings": { "AudioDescriptionBroadcasterMix": "NORMAL", "Bitrate": 160000, "RateControlMode": "CBR", "CodecProfile": "LC", "CodingMode": "CODING_MODE_2_0", "RawFormat": "NONE", "SampleRate": 48000, "Specification": "MPEG4" } }, "LanguageCodeControl": "FOLLOW_INPUT", "AudioType": 0 } ] } ], "OutputGroupSettings": { "Type": "FILE_GROUP_SETTINGS", "FileGroupSettings": { "Destination": "s3://EXAMPLE-BUCKET/MP4/", "DestinationSettings": { "S3Settings": { "AccessControl": { "CannedAcl": "PUBLIC_READ" } } } } } }, { "CustomName": "Thumbnails", "Name": "File Group", "Outputs": [ { "ContainerSettings": { "Container": "RAW" }, "VideoDescription": { "Width": 1280, "ScalingBehavior": "DEFAULT", "Height": 720, "TimecodeInsertion": "DISABLED", "AntiAlias": "ENABLED", "Sharpness": 50, "CodecSettings": { "Codec": "FRAME_CAPTURE", "FrameCaptureSettings": { "FramerateNumerator": 1, "FramerateDenominator": 5, "MaxCaptures": 500, "Quality": 80 } }, "AfdSignaling": "NONE", "DropFrameTimecode": "ENABLED", "RespondToAfd": "NONE", "ColorMetadata": "INSERT" } } ], "OutputGroupSettings": { "Type": "FILE_GROUP_SETTINGS", "FileGroupSettings": { "Destination": "s3://EXAMPLE-BUCKET/Thumbnails/", "DestinationSettings": { "S3Settings": { "AccessControl": { "CannedAcl": "PUBLIC_READ" } } } } } } ], "AdAvailOffset": 0, "Inputs": [ { "AudioSelectors": { "Audio Selector 1": { "Offset": 0, "DefaultSelection": "DEFAULT", "ProgramSelection": 1 } }, "VideoSelector": { "ColorSpace": "FOLLOW" }, "FilterEnable": "AUTO", "PsiControl": "USE_PSI", "FilterStrength": 0, "DeblockFilter": "DISABLED", "DenoiseFilter": "DISABLED", "TimecodeSource": "EMBEDDED", "FileInput": "s3://EXAMPLE-INPUT-BUCKET/input.mp4" } ] } ``` 1. In the `batch-transcode` folder, create a file with a Lambda function. You can use the following Python example and name the file `convert.py`. S3 Batch Operations sends specific task data to a Lambda function and requires result data back. For request and response examples for the Lambda function, information about response and result codes, and example Lambda functions for S3 Batch Operations, see [Invoke AWS Lambda function](batch-ops-invoke-lambda.md). ``` import json import os from urllib.parse import urlparse import uuid import boto3 """ When you run an S3 Batch Operations job, your job invokes this Lambda function. Specifically, the Lambda function is invoked on each video object listed in the manifest that you specify for the S3 Batch Operations job in Step 5. Input parameter "event": The S3 Batch Operations event as a request for the Lambda function. Input parameter "context": Context about the event. Output: A result structure that Amazon S3 uses to interpret the result of the operation. It is a job response returned back to S3 Batch Operations. """ def handler(event, context): invocation_schema_version = event['invocationSchemaVersion'] invocation_id = event['invocationId'] task_id = event['tasks'][0]['taskId'] source_s3_key = event['tasks'][0]['s3Key'] source_s3_bucket = event['tasks'][0]['s3BucketArn'].split(':::')[-1] source_s3 = 's3://' + source_s3_bucket + '/' + source_s3_key result_list = [] result_code = 'Succeeded' result_string = 'The input video object was converted successfully.' # The type of output group determines which media players can play # the files transcoded by MediaConvert. # For more information, see [Creating outputs with AWS Elemental MediaConvert](https://docs.aws.amazon.com/mediaconvert/latest/ug/creating-streaming-and-file-outputs.html). output_group_type_dict = { 'HLS_GROUP_SETTINGS': 'HlsGroupSettings', 'FILE_GROUP_SETTINGS': 'FileGroupSettings', 'CMAF_GROUP_SETTINGS': 'CmafGroupSettings', 'DASH_ISO_GROUP_SETTINGS': 'DashIsoGroupSettings', 'MS_SMOOTH_GROUP_SETTINGS': 'MsSmoothGroupSettings' } try: job_name = 'Default' with open('job.json') as file: job_settings = json.load(file) job_settings['Inputs'][0]['FileInput'] = source_s3 # The path of each output video is constructed based on the values of # the attributes in each object of OutputGroups in the job.json file. destination_s3 = 's3://{0}/{1}/{2}' \ .format(os.environ['amzn-s3-demo-destination-bucket'], os.path.splitext(os.path.basename(source_s3_key))[0], os.path.splitext(os.path.basename(job_name))[0]) for output_group in job_settings['OutputGroups']: output_group_type = output_group['OutputGroupSettings']['Type'] if output_group_type in output_group_type_dict.keys(): output_group_type = output_group_type_dict[output_group_type] output_group['OutputGroupSettings'][output_group_type]['Destination'] = \ "{0}{1}".format(destination_s3, urlparse(output_group['OutputGroupSettings'][output_group_type]['Destination']).path) else: raise ValueError("Exception: Unknown Output Group Type {}." .format(output_group_type)) job_metadata_dict = { 'assetID': str(uuid.uuid4()), 'application': os.environ['Application'], 'input': source_s3, 'settings': job_name } region = os.environ['AWS_DEFAULT_REGION'] endpoints = boto3.client('mediaconvert', region_name=region) \ .describe_endpoints() client = boto3.client('mediaconvert', region_name=region, endpoint_url=endpoints['Endpoints'][0]['Url'], verify=False) try: client.create_job(Role=os.environ['MediaConvertRole'], UserMetadata=job_metadata_dict, Settings=job_settings) # You can customize error handling based on different error codes that # MediaConvert can return. # For more information, see [MediaConvert error codes](https://docs.aws.amazon.com/mediaconvert/latest/ug/mediaconvert_error_codes.html). # When the result_code is TemporaryFailure, S3 Batch Operations retries # the task before the job is completed. If this is the final retry, # the error message is included in the final report. except Exception as error: result_code = 'TemporaryFailure' raise except Exception as error: if result_code != 'TemporaryFailure': result_code = 'PermanentFailure' result_string = str(error) finally: result_list.append({ 'taskId': task_id, 'resultCode': result_code, 'resultString': result_string, }) return { 'invocationSchemaVersion': invocation_schema_version, 'treatMissingKeyAs': 'PermanentFailure', 'invocationId': invocation_id, 'results': result_list } ``` 1. To create a deployment package with `convert.py` and `job.json` as a `.zip` file named `lambda.zip`, in your local terminal, open the `batch-transcode` folder that you created earlier, and run the following command. For **macOS users**, run the following command: ``` zip -r lambda.zip convert.py job.json ``` For **Windows users**, run the following commands: ``` powershell Compress-Archive convert.py lambda.zip ``` ``` powershell Compress-Archive -update job.json lambda.zip ``` ### Create a Lambda function with an execution role (console) 1. Open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. In the left navigation pane, choose **Functions**. 1. Choose **Create function**. 1. Choose **Author from scratch**. 1. Under **Basic information**, do the following: 1. For **Function name**, enter **tutorial-lambda-convert**. 1. For **Runtime**, choose **Python 3.13**. 1. Choose **Change default execution role**, and under **Execution role**, choose **Use an existing role**. 1. Under **Existing role**, choose the name of the IAM role that you created for your Lambda function in [Step 3](#batchops-s3-step3) (for example, **tutorial-lambda-transcode-role**). 1. For the remaining settings, keep the defaults. 1. Choose **Create function**. ### Deploy your Lambda function with .zip file archives and configure the Lambda function (console) 1. In the **Code Source** section of the page for the Lambda function that you created (for example, **tutorial-lambda-convert**), choose **Upload from** and then **.zip file**. 1. Choose **Upload** to select your local `.zip` file. 1. Choose the `lambda.zip` file that you created earlier, and choose **Open**. 1. Choose **Save**. 1. In the **Runtime settings** section, choose **Edit**. 1. To tell the Lambda runtime which handler method in your Lambda function code to invoke, enter **convert.handler** in the **Handler** field. When you configure a function in Python, the value of the handler setting is the file name and the name of the handler module, separated by a dot (`.`). For example, `convert.handler` calls the `handler` method defined in the `convert.py` file. 1. Choose **Save**. 1. On your Lambda function page, choose the **Configuration** tab. In the left navigation pane on the **Configuration** tab, choose **Environment variables**, and then choose **Edit**. 1. Choose **Add environment variable**. Then, enter the specified **Key** and **Value** for each of the following environment variables: + **Key**: **DestinationBucket** **Value**: **amzn-s3-demo-destination-bucket1** This value is the S3 bucket for output media files that you created in [Step 1](#batchops-s3-step1). + **Key**: **MediaConvertRole** **Value**: **arn:aws:iam::*111122223333*:role/tutorial-mediaconvert-role** This value is the ARN of the IAM role for MediaConvert that you created in [Step 2](#batchops-s3-step2). Make sure to replace this ARN with the actual ARN of your IAM role. + **Key**: **Application** **Value**: **Batch-Transcoding** This value is the name of the application. 1. Choose **Save**. 1. (Optional) On the **Configuration** tab, in the **General configuration** section of the left navigation pane, choose **Edit**. In the **Timeout** field, enter **2** min **0** sec. Then, choose **Save**. **Timeout** is the amount of time that Lambda allows a function to run for an invocation before stopping it. The default is 3 seconds. Pricing is based on the amount of memory configured and the amount of time that your code runs. For more information, see [AWS Lambda pricing](https://aws.amazon.com/lambda/pricing/?icmpid=docs_console_unmapped). ## Step 5: Configure Amazon S3 Inventory for your S3 source bucket After setting up the transcoding Lambda function, create an S3 Batch Operations job to transcode a set of videos. First, you need a list of input video objects that you want S3 Batch Operations to run the specified transcoding action on. To get a list of input video objects, you can generate an S3 Inventory report for your S3 source bucket (for example, `amzn-s3-demo-source-bucket`). **Topics** + [ ### Create and configure a bucket for S3 Inventory reports for input videos ](#batchops-s3-step5-bucket) + [ ### Configure Amazon S3 Inventory for your S3 video source bucket ](#batchops-s3-step5-inventory) + [ ### Check the inventory report for your S3 video source bucket ](#batchops-s3-step5-manifest) ### Create and configure a bucket for S3 Inventory reports for input videos To store an S3 Inventory report that lists the objects of the S3 source bucket, create an S3 Inventory destination bucket, and then configure a bucket policy for the bucket to write inventory files to the S3 source bucket. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. Choose **Create bucket**. 1. For **Bucket name**, enter a name for your bucket (for example, `amzn-s3-demo-destination-bucket2`). 1. For **AWS Region**, choose the AWS Region where you want the bucket to reside. The inventory destination bucket must be in the same AWS Region as the source bucket where you are setting up S3 Inventory. The inventory destination bucket can be in a different AWS account. 1. In **Block Public Access settings for this bucket**, keep the default settings (**Block *all *public access** is enabled). 1. For the remaining settings, keep the defaults. 1. Choose **Create bucket**. 1. In the **Buckets** list, choose the name of the bucket that you just created (for example, `amzn-s3-demo-destination-bucket2`). 1. To grant Amazon S3 permission to write data for the inventory reports to the S3 Inventory destination bucket, choose the **Permissions** tab. 1. Scroll down to the **Bucket policy** section, and choose **Edit**. The **Bucket policy** page opens. 1. To grant permissions for S3 Inventory, in the **Policy** field, paste the following bucket policy. Replace the three example values with the following values: + The name of the bucket that you created to store the inventory reports (for example, `amzn-s3-demo-destination-bucket2`). + The name of the source bucket that stores the input videos (for example, `amzn-s3-demo-source-bucket`). + The AWS account ID that you used to create the S3 video source bucket (for example, `111122223333`. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Sid":"InventoryAndAnalyticsExamplePolicy", "Effect":"Allow", "Principal": {"Service": "s3.amazonaws.com"}, "Action":"s3:PutObject", "Resource":["arn:aws:s3:::amzn-s3-demo-destination-bucket2/*"], "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:::amzn-s3-demo-source-bucket" }, "StringEquals": { "aws:SourceAccount": "111122223333", "s3:x-amz-acl": "bucket-owner-full-control" } } } ] } ``` ------ 1. Choose **Save changes**. ### Configure Amazon S3 Inventory for your S3 video source bucket To generate a flat file list of video objects and metadata, you must configure S3 Inventory for your S3 video source bucket. These scheduled inventory reports can include all the objects in the bucket or objects grouped by a shared prefix. In this tutorial, the S3 Inventory report includes all the video objects in your S3 source bucket. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. To configure an S3 Inventory report of the input videos in your S3 source bucket, in the **Buckets** list, choose the name of the S3 source bucket (for example, `amzn-s3-demo-source-bucket`). 1. Choose the **Management** tab. 1. Scroll down to the **Inventory configurations** section, and choose **Create inventory configuration**. 1. For **Inventory configuration name**, enter a name (for example, **tutorial-inventory-config**). 1. Under **Inventory scope**, choose **Current version only** for **Object versions** and keep the other **Inventory scope** settings set to the defaults for this tutorial. 1. In the **Report details** section, for **Destination bucket**, choose **This account**. 1. For **Destination**, choose **Browse S3**, and choose the destination bucket that you created earlier to save the inventory reports to (for example, `amzn-s3-demo-destination-bucket2`). Then choose **Choose path**. The inventory destination bucket must be in the same AWS Region as the source bucket where you are setting up S3 Inventory. The inventory destination bucket can be in a different AWS account. Under the **Destination** bucket field, the **Destination bucket permission** is added to the inventory destination bucket policy, allowing Amazon S3 to place data in the inventory destination bucket. For more information, see [Creating a destination bucket policy](configure-inventory.md#configure-inventory-destination-bucket-policy). 1. For **Frequency**, choose **Daily**. 1. For **Output format**, choose **CSV**. 1. For **Status**, choose **Enable**. 1. In the **Server-side encryption** section, choose **Disable** for this tutorial. For more information, see [Configuring inventory by using the S3 console](configure-inventory.md#configure-inventory-console) and [Granting Amazon S3 permission to use your customer managed key for encryption](configure-inventory.md#configure-inventory-kms-key-policy). 1. In the **Additional fields - *optional*** section, select **Size**, **Last modified**, and **Storage class**. 1. Choose **Create**. For more information, see [Configuring inventory by using the S3 console](configure-inventory.md#configure-inventory-console). ### Check the inventory report for your S3 video source bucket When an inventory report is published, the manifest files are sent to the S3 Inventory destination bucket. 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the name of the video source bucket (for example, `amzn-s3-demo-source-bucket`). 1. Choose **Management**. 1. To see if your S3 Inventory report is ready so that you can create an S3 Batch Operations job in [Step 7](#batchops-s3-step7), under **Inventory configurations**, check whether the **Create job from manifest** button is enabled. **Note** It can take up to 48 hours to deliver the first inventory report. If the **Create job from manifest** button is disabled, the first inventory report has not been delivered. Wait until the first inventory report is delivered and the **Create job from manifest** button is enabled before you create an S3 Batch Operations job in [Step 7](#batchops-s3-step7). 1. To check an S3 Inventory report (`manifest.json`), in the **Destination** column, choose the name of the inventory destination bucket that you created earlier for storing inventory reports (for example, `amzn-s3-demo-destination-bucket2`). 1. On the **Objects** tab, choose the existing folder with the name of your S3 source bucket (for example, `amzn-s3-demo-source-bucket`). Then choose the name that you entered in **Inventory configuration name** when you created the inventory configuration earlier (for example, **tutorial-inventory-config**). You can see a list of folders with the generation dates of the reports as their names. 1. To check the daily S3 Inventory report for a particular date, choose the folder with the corresponding generation date name, and then choose `manifest.json`. 1. To check the details of the inventory report on a specific date, on the **manifest.json** page, choose **Download** or **Open**. ## Step 6: Create an IAM role for S3 Batch Operations To use S3 Batch Operations to do batch-transcoding, you must first create an IAM role to give Amazon S3 permissions to perform S3 Batch Operations. **Topics** + [ ### Create an IAM policy for S3 Batch Operations ](#batchops-s3-step6-policy) + [ ### Create an S3 Batch Operations IAM role and attach permissions policies ](#batchops-s3-step6-role) ### Create an IAM policy for S3 Batch Operations You must create an IAM policy that gives S3 Batch Operations permission to read the input manifest, invoke the Lambda function, and write the S3 Batch Operations job completion report. 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policies**. 1. Choose **Create policy**. 1. Choose the **JSON** tab. 1. In the **JSON** text field, paste the following JSON policy. In the JSON policy, replace the four example values with the following values: + The name of the source bucket that stores your input videos (for example, `amzn-s3-demo-source-bucket`). + The name of the inventory destination bucket that you created in [Step 5](#batchops-s3-step5) to store `manifest.json` files (for example, `amzn-s3-demo-destination-bucket2`). + The name of the bucket that you created in [Step 1](#batchops-s3-step1) to store output media files (for example, `amzn-s3-demo-destination-bucket1`). In this tutorial, we put job completion reports in the destination bucket for output media files. + The role ARN of the Lambda function that you created in [Step 4](#batchops-s3-step4). To find and copy the role ARN of the Lambda function, do the following: + In a new browser tab, open the **Functions** page on the Lambda console at [https://console.aws.amazon.com/lambda/home#/functions](https://console.aws.amazon.com/lambda/home#/functions). + In **Functions** list, choose the name of the Lambda function that you created in [Step 4](#batchops-s3-step4) (for example, **tutorial-lambda-convert**). + Choose **Copy ARN**. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "S3Get", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-source-bucket/*", "arn:aws:s3:::amzn-s3-demo-destination-bucket2/*" ] }, { "Sid": "S3PutJobCompletionReport", "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::amzn-s3-demo-destination-bucket1/*" }, { "Sid": "S3BatchOperationsInvokeLambda", "Effect": "Allow", "Action": [ "lambda:InvokeFunction" ], "Resource": [ "arn:aws:lambda:us-west-2:111122223333:function:tutorial-lambda-convert" ] } ] } ``` ------ 1. Choose **Next: Tags**. 1. Choose **Next: Review**. 1. In the **Name** field, enter **tutorial-s3batch-policy**. 1. Choose **Create policy**. ### Create an S3 Batch Operations IAM role and attach permissions policies 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**, and then choose **Create role**. 1. Choose the **AWS service** role type, and then choose the **S3** service. 1. Under **Select your use case**, choose **S3 Batch Operations**. 1. Choose **Next**. 1. Under **Attach permissions**, enter the name of the IAM policy that you created earlier (for example, **tutorial-s3batch-policy**) in the search box to filter the list of policies. Select the check box next to the name of the policy (for example, **tutorial-s3batch-policy**). 1. Choose **Next**. 1. For **Role name**, enter **tutorial-s3batch-role**. 1. Choose **Create role**. After you create the IAM role for S3 Batch Operations, the following trust policy is automatically attached to the role. This trust policy allows the S3 Batch Operations service principal to assume the IAM role. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Principal":{ "Service":"batchoperations.s3.amazonaws.com" }, "Action":"sts:AssumeRole" } ] } ``` ------ ## Step 7: Create and run an S3 Batch Operations job To create an S3 Batch Operations job to process the input videos in your S3 source bucket, you must specify parameters for this particular job. **Note** Before you start creating an S3 Batch Operations job, make sure that the **Create job from manifest** button is enabled. For more information, see [Check the inventory report for your S3 video source bucket](#batchops-s3-step5-manifest). If the **Create job from manifest** button is disabled, the first inventory report has not been delivered and you must wait until the button is enabled. After you configure Amazon S3 Inventory for your S3 source bucket in [Step 5](#batchops-s3-step5), it can take up to 48 hours to deliver the first inventory report. **Topics** + [ ### Create an S3 Batch Operations job ](#batchops-s3-step7-create-job) + [ ### Run the S3 Batch Operations job to invoke your Lambda function ](#batchops-s3-step7-run-job) + [ ### (Optional) Check your completion report ](#batchops-s3-step7-check-report) + [ ### (Optional) Monitor each Lambda invocation in the Lambda console ](#batchops-s3-step7-monitor-lambda) + [ ### (Optional) Monitor each MediaConvert video-transcoding job in the MediaConvert console ](#batchops-s3-step7-monitor-mediaconvert) ### Create an S3 Batch Operations job 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Batch Operations**. 1. Choose **Create job**. 1. For **AWS Region**, choose the Region where you want to create your job. In this tutorial, to use the S3 Batch Operations job to invoke a Lambda function, you must create the job in the same Region as the S3 video source bucket where the objects referenced in the manifest are located. 1. In the **Manifest** section, do the following: 1. For **Manifest format**, choose **S3 Inventory report (manifest.json)**. 1. For **Manifest object**, choose **Browse S3** to find the bucket that you created in [Step 5](#batchops-s3-step5) for storing inventory reports (for example, `amzn-s3-demo-destination-bucket2`). On the **Manifest object ** page, navigate through the object names until you find a `manifest.json` file for a specific date. This file lists the information about all the videos that you want to batch-transcode. When you've found the `manifest.json` file that you want to use, choose the option button next to it. Then choose **Choose path**. 1. (Optional) For **Manifest object version ID - *optional***, enter the version ID for the manifest object if you want to use a version other than the most recent. 1. Choose **Next**. 1. To use the Lambda function to transcode all the objects listed in the selected `manifest.json` file, under **Operation type**, choose **Invoke AWS Lambda function**. 1. In the **Invoke Lambda function** section, do the following: 1. Choose **Choose from functions in your account**. 1. For **Lambda function**, choose the Lambda function that you created in [Step 4](#batchops-s3-step4) (for example, **tutorial-lambda-convert**). 1. For **Lambda function version**, keep the default value **\$1LATEST**. 1. Choose **Next**. The **Configure additional options** page opens. 1. In the **Additional options** section, keep the default settings. For more information about these options, see [Batch Operations job request elements](batch-ops-create-job.md#batch-ops-create-job-request-elements). 1. In the **Completion report** section, for **Path to completion report destination**, choose **Browse S3**. Find the bucket that you created for output media files in [Step 1](#batchops-s3-step1) (for example, `amzn-s3-demo-destination-bucket1`). Choose the option button next to that bucket's name. Then choose **Choose path**. For the remaining **Completion report** settings, keep the defaults. For more information about completion report settings, see [Batch Operations job request elements](batch-ops-create-job.md#batch-ops-create-job-request-elements). A completion report maintains a record of the job's details and the operations performed. 1. In the **Permissions** section, choose **Choose from existing IAM roles**. For **IAM role**, choose the IAM role for your S3 Batch Operations job that you created in [Step 6](#batchops-s3-step6) (for example, **tutorial-s3batch-role**). 1. Choose **Next**. 1. On the **Review** page, review the settings. Then choose **Create job**. After S3 finishes reading your S3 Batch Operations job's manifest, it sets the **Status** of the job to **Awaiting your confirmation to run**. To see updates to the job's status, refresh the page. You can't run your job until its status is **Awaiting your confirmation to run**. ### Run the S3 Batch Operations job to invoke your Lambda function Run your Batch Operations job to invoke your Lambda function for video transcoding. If your job fails, you can check your completion report to identify the cause. **To run the S3 Batch Operations job** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Batch Operations**. 1. In the **Jobs** list, choose the **Job ID** of the job on the first row, which is the S3 Batch Operations job that you created earlier. 1. Choose **Run job**. 1. Review your job parameters again, and confirm that the value for **Total objects listed in manifest** is the same as the number of objects in the manifest. Then choose **Run job**. Your S3 Batch Operations job page opens. 1. After the job starts running, on your job page, under **Status**, check the progress of your S3 Batch Operations job, such as **Status**, **% Complete**, **Total succeeded (rate)**, **Total failed (rate)**, **Date terminated**, and **Reason for termination**. When the S3 Batch Operations job completes, view the data on your job page to confirm that the job finished as expected. If more than 50 percent of an S3 Batch Operations job's object operations fail after more than 1,000 operations have been attempted, the job automatically fails. To check your completion report to identify the cause of the failures, use the following optional procedure. ### (Optional) Check your completion report You can use your completion report to determine which objects failed and the cause of the failures. **To check your completion report for details about failed objects** 1. On the page of your S3 Batch Operations job, scroll down to the **Completion report** section, and choose the link under **Completion report destination**. The S3 output destination bucket's page opens. 1. On the **Objects** tab, choose the folder that has a name ending with the job ID of the S3 Batch Operations job that you created earlier. 1. Choose **results/**. 1. Select the check box next to the `.csv` file. 1. To view the job report, choose **Open** or **Download**. ### (Optional) Monitor each Lambda invocation in the Lambda console After the S3 Batch Operations job starts running, the job invokes the Lambda function for each input video object. S3 writes logs of each Lambda invocation to CloudWatch Logs. You can use the Lambda console's monitoring dashboard to monitor your Lambda function. 1. Open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. In the left navigation pane, choose **Functions**. 1. In the **Functions** list, choose the name of the Lambda function that you created in [Step 4](#batchops-s3-step4) (for example, **tutorial-lambda-convert**). 1. Choose the **Monitor** tab. 1. Under **Metrics**, see the runtime metrics for your Lambda function. 1. Under **Logs**, view log data for each Lambda invocation through CloudWatch Logs Insights. **Note** When you use S3 Batch Operations with a Lambda function, the Lambda function is invoked on each object. If your S3 Batch Operations job is large, it can invoke multiple Lambda functions at the same time, causing a spike in Lambda concurrency. Each AWS account has a Lambda concurrency quota per Region. For more information, see [AWS Lambda Function Scaling](https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html) in the *AWS Lambda Developer Guide*. A best practice for using Lambda functions with S3 Batch Operations is to set a concurrency limit on the Lambda function itself. Setting a concurrency limit keeps your job from consuming most of your Lambda concurrency and potentially throttling other functions in your account. For more information, see [Managing Lambda reserved concurrency](https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html) in the *AWS Lambda Developer Guide*. ### (Optional) Monitor each MediaConvert video-transcoding job in the MediaConvert console A MediaConvert job does the work of transcoding a media file. When your S3 Batch Operations job invokes your Lambda function for each video, each Lambda function invocation creates a MediaConvert transcoding job for each input video. 1. Sign in to the AWS Management Console and open the MediaConvert console at [https://console.aws.amazon.com/mediaconvert/](https://console.aws.amazon.com/mediaconvert/). 1. If the MediaConvert introductory page appears, choose **Get started**. 1. From the list of **Jobs**, view each row to monitor the transcoding task for each input video. 1. Identify the row of a job that you want to check, and choose the **Job ID** link to open the job details page. 1. On the **Job summary** page, under **Outputs**, choose the link for the HLS, MP4, or Thumbnails output, depending on what is supported by your browser, to go to the S3 destination bucket for the output media files. 1. In the corresponding folder (HLS, MP4, or Thumbnails) of your S3 output destination bucket, choose the name of the output media file object. The object's detail page opens. 1. On the object's detail page, under **Object overview**, choose the link under **Object URL** to watch the transcoded output media file. ## Step 8: Check the output media files from your S3 destination bucket **To check the output media files from your S3 destination bucket** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the name of the S3 destination bucket for output media files that you created in [Step 1](#batchops-s3-step1) (for example, `amzn-s3-demo-destination-bucket1`). 1. On the **Objects** tab, each input video has a folder that has the name of the input video. Each folder contains the transcoded output media files for an input video. To check the output media files for an input video, do the following: 1. Choose the folder with the name of the input video that you want to check. 1. Choose the **Default/** folder. 1. Choose the folder for a transcoded format (HLS, MP4, or thumbnails in this tutorial). 1. Choose the name of the output media file. 1. To watch the transcoded file, on the object's details page, choose the link under **Object URL**. Output media files in the HLS format are split into short segments. To play these videos, embed the object URL of the `.m3u8` file in a compatible player. ## Step 9: Clean up If you transcoded videos using S3 Batch Operations, Lambda, and MediaConvert only as a learning exercise, delete the AWS resources that you allocated so that you no longer accrue charges. **Topics** + [ ### Delete the S3 Inventory configuration for your S3 source bucket ](#batchops-s3-step9-delete-inventory) + [ ### Delete the Lambda function ](#batchops-s3-step9-delete-lambda) + [ ### Delete the CloudWatch log group ](#batchops-s3-step9-delete-cw) + [ ### Delete the IAM roles together with the inline policies for the IAM roles ](#batchops-s3-step9-delete-role) + [ ### Delete the customer-managed IAM policy ](#batchops-s3-step9-delete-policy) + [ ### Empty the S3 buckets ](#batchops-s3-step9-empty-bucket) + [ ### Delete the S3 buckets ](#batchops-s3-step9-delete-bucket) ### Delete the S3 Inventory configuration for your S3 source bucket 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the name of your source bucket (for example, `amzn-s3-demo-source-bucket`). 1. Choose the **Management** tab. 1. In the **Inventory configurations** section, choose the option button next to the inventory configuration that you created in [Step 5](#batchops-s3-step5) (for example, **tutorial-inventory-config**). 1. Choose **Delete**, and then choose **Confirm**. ### Delete the Lambda function 1. Open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. In the left navigation pane, choose **Functions**. 1. Select the check box next to the function that you created in [Step 4](#batchops-s3-step4) (for example, **tutorial-lambda-convert**). 1. Choose **Actions**, and then choose **Delete**. 1. In the **Delete function** dialog box, choose **Delete**. ### Delete the CloudWatch log group 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the left navigation pane, choose **Logs**, and then choose **Log groups**. 1. Select the check box next to the log group that has a name ending with the Lambda function that you created in [Step 4](#batchops-s3-step4) (for example, **tutorial-lambda-convert**). 1. Choose **Actions**, and then choose **Delete log group(s)**. 1. In the **Delete log group(s)** dialog box, choose **Delete**. ### Delete the IAM roles together with the inline policies for the IAM roles To delete the IAM roles that you created in [Step 2](#batchops-s3-step2), [Step 3](#batchops-s3-step3), and [Step 6](#batchops-s3-step6), do the following: 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Roles**, and then select the check boxes next to the role names that you want to delete. 1. At the top of the page, choose **Delete**. 1. In the confirmation dialog box, enter the required response in the text input field based on the prompt, and choose **Delete**. ### Delete the customer-managed IAM policy To delete the customer-managed IAM policy that you created in [Step 6](#batchops-s3-step6), do the following: 1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). 1. In the left navigation pane, choose **Policies**. 1. Choose the option button next to the policy that you created in [Step 6](#batchops-s3-step6) (for example, **tutorial-s3batch-policy**). You can use the search box to filter the list of policies. 1. Choose **Actions**, and then choose **Delete**. 1. Confirm that you want to delete this policy by entering its name in the text field, and then choose **Delete**. ### Empty the S3 buckets To empty the S3 buckets that you created in [Prerequisites](#batchops-s3-prerequisites), [Step 1](#batchops-s3-step1), and [Step 5](#batchops-s3-step5), do the following: 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the option button next to the name of the bucket that you want to empty, and then choose **Empty**. 1. On the **Empty bucket** page, confirm that you want to empty the bucket by entering **permanently delete** in the text field, and then choose **Empty**. ### Delete the S3 buckets To delete the S3 buckets that you created in [Prerequisites](#batchops-s3-prerequisites), [Step 1](#batchops-s3-step1), and [Step 5](#batchops-s3-step5), do the following: 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. In the **Buckets** list, choose the option button next to the name of the bucket that you want to delete. 1. Choose **Delete**. 1. On the **Delete bucket** page, confirm that you want to delete the bucket by entering the bucket name in the text field, and then choose **Delete bucket**. ## Next steps After completing this tutorial, you can further explore other relevant use cases: + You can use Amazon CloudFront to stream the transcoded media files to viewers across the globe. For more information, see [Tutorial: Hosting on-demand streaming video with Amazon S3, Amazon CloudFront, and Amazon Route 53](tutorial-s3-cloudfront-route53-video-streaming.md). + You can transcode videos at the moment when you upload them to the S3 source bucket. To do so, you can configure an Amazon S3 event trigger that automatically invokes the Lambda function to transcode new objects in S3 with MediaConvert. For more information, see [Tutorial: Using an Amazon S3 trigger to invoke a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html) in the *AWS Lambda Developer Guide*. # Troubleshooting S3 Batch Operations Amazon S3 Batch Operations allows you to perform large-scale operations on Amazon S3 objects. This guide helps you troubleshoot common issues you might encounter. To troubleshoot issues with S3 Batch Replication, see [Troubleshooting replication](replication-troubleshoot.md). There are two primary types of failures that result in Batch operation errors: 1. **API Failure** – The requested API (such as `CreateJob`) has failed to execute. 1. **Job Failure** – The initial API request succeeded but the job failed, for example, due to issues with the manifest or permissions to objects specified in the manifest. ## NoSuchJobException **Type**: API failure The `NoSuchJobException` occurs when S3 Batch Operations cannot locate the specified job. This error can happen in several scenarios beyond simple job expiration. Common causes include the following. 1. **Job expiration** – Jobs are automatically deleted 90 days after reaching a terminal state (`Complete`, `Cancelled`, or `Failed`). 1. **Incorrect job ID** – The job ID used in `DescribeJob` or `UpdateJobStatus` doesn't match the ID returned by `CreateJob`. 1. **Wrong region** – Attempting to access a job in a different region than where it was created. 1. **Wrong account** – Using a job ID from a different AWS account. 1. **Job ID format errors** – Typos, extra characters, or incorrect formatting in the job ID. 1. **Timing issues** – Checking job status immediately after creation before the job is fully registered. Related error messages include the following. 1. `No such job` 1. `The specified job does not exist` ### Best practices to prevent `NoSuchJobException` API failures 1. **Store job IDs immediately** – Save the job ID from the `CreateJob` response before making subsequent API calls. 1. **Implement retry logic** – Add exponential backoff when checking job status immediately after creation. 1. **Set up monitoring** – Create CloudWatch alarms to track job completion before the 90-day expiration. For details, see [Using CloudWatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) in the Amazon CloudWatch User Guide. 1. **Use consistent regions** – Ensure all job operations use the same region as job creation. 1. **Validate input** – Check job ID format before making API calls. ### When jobs expire Jobs in terminal states are automatically deleted after 90 days. To avoid losing job information, consider the following. 1. **Download completion reports before expiration** – For instructions on retrieving and storing job results, see [Completion reports](batch-ops-job-status.md#batch-ops-completion-report). 1. **Archive job metadata in your own systems** – Store critical job information in your databases or monitoring systems. 1. **Set up automated notifications before the 90-day deadline** – Use Amazon EventBridge to create rules that trigger notifications when jobs complete. For more information, see [Amazon S3 Event Notifications](EventNotifications.md). ### `NoSuchJobException` troubleshooting 1. Use the following command to verify the job exists in your account and region. ``` aws s3control list-jobs --account-id 111122223333 --region us-east-1 ``` 1. Use the following command to search across all job statuses. Possible job statuses include `Active`, `Cancelled`, `Cancelling`, `Complete`, `Completing`, `Failed`, `Failing`, `New`, `Paused`, `Pausing`, `Preparing`, `Ready`, and `Suspended`. ``` aws s3control list-jobs --account-id 111122223333 --job-statuses your-job-status ``` 1. Use the following command to check if the job exists in other regions where you commonly create jobs. ``` aws s3control list-jobs --account-id 111122223333 --region job-region-1 aws s3control list-jobs --account-id 111122223333 --region job-region-2 ``` 1. Validate the job ID format. Job IDs typically contain 36 character, such as `12345678-1234-1234-1234-123456789012`. Check for extra spaces, missing characters, or case sensitivity issues and verify you're using the complete job ID returned by the `CreateJob` command. 1. Use the following command to check CloudTrail logs for job creation events. ``` aws logs filter-log-events --log-group-name CloudTrail/S3BatchOperations \ --filter-pattern "{ $.eventName = CreateJob }" \ --start-time timestamp ``` ### AccessDeniedException **Type**: API failure The `AccessDeniedException` occurs when an S3 Batch Operations request is blocked due to insufficient permissions, unsupported operations, or policy restrictions. This is one of the most common errors in Batch Operations. It has the following common causes: 1. **Missing IAM permissions** – The IAM identity lacks required permissions for Batch Operations APIs. 1. **Insufficient S3 permissions** – Missing permissions to access source or destination buckets and objects. 1. **Job execution role issues** – The job execution role lacks permissions to perform the specified operation. 1. **Unsupported operations** – Attempting to use operations not supported in the current region or bucket type. 1. **Cross-account access issues** – Missing permissions for cross-account bucket or object access. 1. **Resource-based policy restrictions** – Bucket policies or object ACLs blocking the operation. 1. **Service Control Policy (SCP) restrictions** – Organization-level policies preventing the operation. Related error messages: 1. `Access Denied` 1. `User: arn:aws:iam::account:user/username is not authorized to perform: s3:operation` 1. `Cross-account pass role is not allowed` 1. `The bucket policy does not allow the specified operation` #### Best practices to prevent AccessDeniedException API failures 1. **Use least privilege principle** – Grant only the minimum permissions required for your specific operations. 1. **Test permissions before large jobs** – Run small test jobs to validate permissions before processing thousands of objects. 1. **Use IAM policy simulator** – Test policies before deployment using the IAM policy simulator. For more information, see [IAM policy testing with the IAM policy simulator](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html) in the IAM User Guide. 1. **Implement proper cross-account setup** – Check your cross-account access configuration for cross-account job configurations. For more information, see [IAM tutorial: Delegate access across AWS accounts using IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html) in the IAM User Guide. 1. **Monitor permission changes** – Set up CloudTrail alerts for IAM policy modifications that might affect Batch Operations. 1. **Document role requirements** – Maintain clear documentation of required permissions for each job type. 1. **Use common permission templates** - Use the permission examples and policy templates: 1. [Granting permissions for Batch Operations](batch-ops-iam-role-policies.md) 1. [Cross account resources in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies-cross-account-resource-access.html) in the IAM User Guide. 1. [Control access to VPC endpoints using endpoint policies](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html) in the AWS PrivateLink Guide. #### AccessDeniedException troubleshooting Follow these steps systematically to identify and resolve permission issues. 1. Check [Operations supported by S3 Batch Operations](batch-ops-operations.md) for supported operations by region. Confirm directory bucket operations are only available at Regional and Zonal endpoints. Verify the operation is supported for your bucket's storage class. 1. Use the following command to determine if you can list jobs. ``` aws s3control list-jobs --account-id 111122223333 ``` 1. Use the following command to check IAM permissions for the requesting identity. The account running the job needs the following permissions: `s3:CreateJob`, `s3:DescribeJob`, `s3:ListJobs`, `s3:UpdateJobPriority`, `s3:UpdateJobStatus`, and `iam:PassRole`. ``` aws sts get-caller-identity 111122223333 ``` 1. Use the following command to check if the role exists and is assumable. ``` aws iam get-role --role-name role-name ``` 1. Use the following command to review the role's trust policy. The role running the job must have the following: 1. A trust relationship allowing `batchoperations.s3.amazonaws.com` to assume the role. 1. The operations the batch operation is performing (such as `s3:PutObjectTagging` for tagging operations). 1. Access to the source and destination buckets. 1. Permission to read the manifest file. 1. Permission to write completion reports. ``` aws iam get-role --role-name role-name --query 'Role.AssumeRolePolicyDocument' ``` 1. Use the following command to test access to the manifest and source buckets. ``` aws s3 ls s3://amzn-s3-demo-bucket ``` 1. Test the operation being performed by the batch operation. For example, if the batch operation performs tagging, tag a sample object in the source bucket. 1. Review bucket policies for policies that might deny the operation. 1. Check object ACLs if working with legacy access controls. 1. Verify no Service Control Policies (SCPs) are blocking the operation. 1. Confirm VPC endpoint policies allow Batch Operations if using VPC endpoints. 1. Use the following command to use CloudTrail to identify permission failures. ``` aws logs filter-log-events --log-group-name CloudTrail/S3BatchOperations \ --filter-pattern "{ $.errorCode = AccessDenied }" \ --start-time timestamp ``` #### SlowDownError **Type**: API failure The `SlowDownError` exception occurs when your account has exceeded the request rate limit for S3 Batch Operations APIs. This is a throttling mechanism to protect the service from being overwhelmed by too many requests. It has the following common causes: 1. **High API request frequency** – Making too many API calls in a short time period. 1. **Concurrent job operations** – Multiple applications or users creating/managing jobs simultaneously. 1. **Automated scripts without rate limiting** – Scripts that don't implement proper backoff strategies. 1. **Polling job status too frequently** – Checking job status more often than necessary. 1. **Burst traffic patterns** – Sudden spikes in API usage during peak processing times. 1. **Regional capacity limits** – Exceeding the allocated request capacity for your region. Related error messages: 1. `SlowDown` 1. `Please reduce your request rate` 1. `Request rate exceeded` #### Best practices to prevent SlowDownError API failures 1. **Implement client-side rate limiting** – Add delays between API calls in your applications. 1. **Use exponential backoff with jitter** – Randomize retry delays to avoid thundering herd problems. 1. **Set up proper retry logic** – Implement automatic retries with increasing delays for transient errors. 1. **Use event-driven architectures** – Replace polling with EventBridge notifications for job status changes. 1. **Distribute load across time** – Stagger job creation and status checks across different time periods. 1. **Monitor and alert on rate limits** – Set up CloudWatch alarms to detect when you're approaching limits. Most AWS SDKs include built-in retry logic for rate limiting errors. Configure them as follows: 1. **AWS CLI** – Use `cli-read-timeout` and `cli-connect-timeout` parameters. 1. **AWS SDK for Python (Boto3)** – Configure retry modes and max attempts in your client configuration. 1. **AWS SDK for Java** – Use `RetryPolicy` and `ClientConfiguration` settings. 1. **AWS SDK for JavaScript** – Configure `maxRetries` and `retryDelayOptions`. For more information about retry patterns and best practices, see [Retry with backoff pattern](https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html) in the AWS Prescriptive Guidance guide. #### SlowDownError troubleshooting 1. In your code, implement exponential backoff immediately. **Example of exponential backoff in bash** ``` for attempt in {1..5}; do if aws s3control describe-job --account-id 111122223333 --job-id job-id; then break else wait_time=$((2**attempt)) echo "Rate limited, waiting ${wait_time} seconds..." sleep $wait_time fi done ``` 1. Use CloudTrail to identify the source of high request volume. ``` aws logs filter-log-events \ --log-group-name CloudTrail/S3BatchOperations \ --filter-pattern "{ $.eventName = CreateJob || $.eventName = DescribeJob }" \ --start-time timestamp \ --query 'events[*].[eventTime,sourceIPAddress,userIdentity.type,eventName]' ``` 1. Review polling frequency. 1. Avoid checking job status more than once every 30 seconds for active jobs. 1. Use job completion notifications instead of polling when possible. 1. Implement jitter in your polling intervals to avoid synchronized requests. 1. Optimize your API usage patterns. 1. Batch multiple operations when possible. 1. Use `ListJobs` to get status of multiple jobs in one call. 1. Cache job information to reduce redundant API calls. 1. Spread job creation across time rather than creating many jobs simultaneously. 1. Use CloudWatch metrics for API calls to monitor your request patterns. ``` aws logs put-metric-filter \ --log-group-name CloudTrail/S3BatchOperations \ --filter-name S3BatchOpsAPICallCount \ --filter-pattern "{ $.eventSource = s3.amazonaws.com && $.eventName = CreateJob }" \ --metric-transformations \ metricName=S3BatchOpsAPICalls,metricNamespace=Custom/S3BatchOps,metricValue=1 ``` ## InvalidManifestContent **Type**: Job failure The `InvalidManifestContent` exception occurs when there are issues with the manifest file format, content, or structure that prevent S3 Batch Operations from processing the job. It has the following common causes: 1. **Format violations** – Missing required columns, incorrect delimiters, or malformed CSV structure. 1. **Content encoding issues** – Incorrect character encoding, BOM markers, or non-UTF-8 characters. 1. **Object key problems** – Invalid characters, improper URL encoding, or keys exceeding length limits. 1. **Size limitations** – Manifest contains more objects than the operation supports. 1. **Version ID format errors** – Malformed or invalid version IDs for versioned objects. 1. **ETag format issues** – Incorrect ETag format or missing quotes for operations that require ETags. 1. **Inconsistent data** – Mixed formats within the same manifest or inconsistent column counts. Related error messages: 1. `Required fields are missing in the schema: + missingFields` 1. `Invalid Manifest Content` 1. `The S3 Batch Operations job failed because it contains more keys than the maximum allowed in a single job` 1. `Invalid object key format` 1. `Manifest file is not properly formatted` 1. `Invalid version ID format` 1. `ETag format is invalid` ### Best practices to prevent InvalidManifestContent job failures 1. **Validate before upload** – Test manifest format with small jobs before processing large datasets. 1. **Use consistent encoding** – Always use UTF-8 encoding without BOM for manifest files. 1. **Implement manifest generation standards** – Create templates and validation procedures for manifest creation. 1. **Handle special characters properly** – URL encode object keys that contain special characters. 1. **Monitor object counts** – Track manifest size and split large jobs proactively. 1. **Validate object existence** – Verify objects exist before including them in manifests. 1. **Use AWS tools for manifest generation** – Leverage AWS CLI `s3api list-objects-v2` to generate properly formatted object lists. Common manifest issues and solutions: 1. **Missing required columns** – Ensure your manifest includes all required columns for your operation type. The most common missing columns are Bucket and Key. 1. **Incorrect CSV format** – Use comma delimiters, ensure consistent column counts across all rows, and avoid embedded line breaks within fields. 1. **Special characters in object keys** – URL encode object keys that contain spaces, Unicode characters, or XML special characters (<, >, &, ", '). 1. **Large manifest files** – Split manifests with more than the operation limit into multiple smaller manifests and create separate jobs. 1. **Invalid version IDs** – Ensure version IDs are properly formatted alphanumeric strings. Remove version ID column if not needed. 1. **Encoding issues** – Save manifest files as UTF-8 without BOM. Avoid copying manifests through systems that might alter encoding. For detailed manifest format specifications and examples, see the following: 1. [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest) 1. [Operations supported by S3 Batch Operations](batch-ops-operations.md) 1. [Naming Amazon S3 objects](object-keys.md) ### InvalidManifestContent troubleshooting 1. Download and inspect the manifest file. Manually verify the manifest meets format requirements: 1. CSV format with comma delimiters. 1. UTF-8 encoding without BOM. 1. Consistent number of columns across all rows. 1. No empty lines or trailing spaces. 1. Object keys properly URL encoded if they contain special characters. Use the following command to download the manifest file. ``` aws s3 cp s3://amzn-s3-demo-bucket1/manifest-key ./manifest.csv ``` 1. Check required columns for your operation: 1. All operations: `Bucket`, `Key` 1. Copy operations: `VersionId` (optional) 1. Restore operations: `VersionId` (optional) 1. Replace tag operations: No additional columns required. 1. Replace ACL operations: No additional columns required. 1. Initiate restore: `VersionId` (optional) 1. Check object count limits: 1. Copy: 1 billion objects maximum. 1. Delete: 1 billion objects maximum. 1. Restore: 1 billion objects maximum. 1. Tagging: 1 billion objects maximum. 1. ACL: 1 billion objects maximum. 1. Create a test manifest with a few objects from your original manifest. 1. Use the following command to check if a sample of objects from the manifest exist. ``` aws s3 ls s3://amzn-s3-demo-bucket1/object-key ``` 1. Check job failure details and review the failure reason and any specific error details in the job description. ``` aws s3control describe-job --account-id 111122223333 --job-id job-id ``` # Querying data in place with Amazon S3 Select **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) With Amazon S3 Select, you can use structured query language (SQL) statements to filter the contents of an Amazon S3 object and retrieve only the subset of data that you need. By using Amazon S3 Select to filter this data, you can reduce the amount of data that Amazon S3 transfers, which reduces the cost and latency to retrieve this data. Amazon S3 Select only allows you to query one object at a time. It works on an object stored in CSV, JSON, or Apache Parquet format. It also works with an object that is compressed with GZIP or BZIP2 (for CSV and JSON objects only), and a server-side encrypted object. You can specify the format of the results as either CSV or JSON, and you can determine how the records in the result are delimited. You pass SQL expressions to Amazon S3 in the request. Amazon S3 Select supports a subset of SQL. For more information about the SQL elements that are supported by Amazon S3 Select, see [SQL reference for Amazon S3 Select](s3-select-sql-reference.md). You can perform SQL queries by using the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the `SelectObjectContent` REST API operation, or the AWS SDKs. **Note** The Amazon S3 console limits the amount of data returned to 40 MB. To retrieve more data, use the AWS CLI or the API. ## Requirements and limits The following are requirements for using Amazon S3 Select: + You must have `s3:GetObject` permission for the object you are querying. + If the object you are querying is encrypted with server-side encryption with customer-provided keys (SSE-C), you must use `https`, and you must provide the encryption key in the request. The following limits apply when using Amazon S3 Select: + S3 Select can query only one object per request. + S3 Select supports querying files up to 5 TB in size. + The maximum length of a SQL expression is 256 KB. + The maximum length of a record in the input or result is 1 MB. + Amazon S3 Select can only emit nested data by using the JSON output format. + You cannot query an object stored in the S3 Glacier Flexible Retrieval, S3 Glacier Deep Archive, or Reduced Redundancy Storage (RRS) storage classes. You also cannot query an object stored in the S3 Intelligent-Tiering Archive Access tier or the S3 Intelligent-Tiering Deep Archive Access tier. For more information about storage classes, see [Understanding and managing Amazon S3 storage classes](storage-class-intro.md). Additional limitations apply when using Amazon S3 Select with a Parquet object: + Amazon S3 Select supports only columnar compression using GZIP or Snappy. Amazon S3 Select doesn't support whole-object compression for a Parquet object. + Amazon S3 Select doesn't support Parquet output. You must specify the output format as CSV or JSON. + The maximum uncompressed row group size is 512 MB. + You must use the data types that are specified in the object's schema. + Selecting on a repeated field returns only the last value. ## Constructing a request When you construct a request, you provide details of the object that is being queried by using an `InputSerialization` object. You provide details of how the results are to be returned by using an `OutputSerialization` object. You also include the SQL expression that Amazon S3 uses to filter the request. For more information about constructing an Amazon S3 Select request, see [https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html) in the *Amazon Simple Storage Service API Reference*. You can also see one of the SDK code examples in the following sections. ### Requests using scan ranges With Amazon S3 Select, you can scan a subset of an object by specifying a range of bytes to query. This capability lets you parallelize scanning the whole object by splitting the work into separate Amazon S3 Select requests for a series of non-overlapping scan ranges. Scan ranges don't need to be aligned with record boundaries. An Amazon S3 Select scan range request runs across the byte range that you specify. A record that starts within the specified scan range but extends beyond that scan range will be processed by the query. For example, the following shows an Amazon S3 object that contains a series of records in a line-delimited CSV format: ``` A,B C,D D,E E,F G,H I,J ``` Suppose that you're using the Amazon S3 Select `ScanRange` parameter and *Start* at (Byte) 1 and *End* at (Byte) 4. So the scan range would start at "`,`" and scan until the end of the record starting at `C`. Your scan range request will return the result `C, D` because that is the end of the record. Amazon S3 Select scan range requests support Parquet, CSV (without quoted delimiters), or JSON objects (in `LINES` mode only). CSV and JSON objects must be uncompressed. For line-based CSV and JSON objects, when a scan range is specified as part of the Amazon S3 Select request, all records that start within the scan range are processed. For Parquet objects, all of the row groups that start within the scan range requested are processed. Amazon S3 Select scan range requests are available to use with the AWS CLI, Amazon S3 API, and AWS SDKs. You can use the `ScanRange` parameter in the Amazon S3 Select request for this feature. For more information, see [https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html](https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html) in the *Amazon Simple Storage Service API Reference*. ## Errors Amazon S3 Select returns an error code and associated error message when an issue is encountered while attempting to run a query. For a list of error codes and descriptions, see the [List of SELECT Object Content Error Codes](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#SelectObjectContentErrorCodeList) section of the *Error Responses* page in the *Amazon Simple Storage Service API Reference*. For more information about Amazon S3 Select, see the following topics. **Topics** + [ ## Requirements and limits ](#selecting-content-from-objects-requirements-and-limits) + [ ## Constructing a request ](#selecting-content-from-objects-contructing-request) + [ ## Errors ](#selecting-content-from-objects-errors) + [ # Examples of using Amazon S3 Select on an object ](using-select.md) + [ # SQL reference for Amazon S3 Select ](s3-select-sql-reference.md) # Examples of using Amazon S3 Select on an object **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) You can use S3 Select to select content from one object by using the Amazon S3 console, the REST API, and the AWS SDKs. For more information about supported SQL functions for S3 Select, see [SQL functions](s3-select-sql-reference-sql-functions.md). ## Using the S3 console **To select content from an object in the Amazon S3 console** 1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/). 1. In the left navigation pane, choose **Buckets**. 1. Choose the bucket that contains the object that you want to select content from, and then choose the name of the object. 1. Choose **Object actions**, and choose **Query with S3 Select**. 1. Configure **Input settings**, based on the format of your input data. 1. Configure **Output settings**, based on the format of the output that you want to receive. 1. To extract records from the chosen object, under **SQL query**, enter the SELECT SQL commands. For more information on how to write SQL commands, see [SQL reference for Amazon S3 Select](s3-select-sql-reference.md). 1. After you enter SQL queries, choose **Run SQL query**. Then, under **Query results**, you can see the results of your SQL queries. ## Using the REST API You can use the AWS SDKs to select content from an object. However, if your application requires it, you can send REST requests directly. For more information about the request and response format, see [https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html). ## Using the AWS SDKs You can use Amazon S3 Select to select some of the content of an object by using the `selectObjectContent` method. If this method is successful, it returns the results of the SQL expression. ------ #### [ Java ] To use Amazon S3 Select with the AWS SDK for Java, you can return the value of the first column for each record that is stored in an object that contains data stored in CSV format. You can also request `Progress` and `Stats` messages to be returned. You must provide a valid bucket name and an object that contains data in CSV format. To use Amazon S3 Select with the AWS SDK for Java, you can return the value of the first column for each record that is stored in an object that contains data stored in CSV format. You can also request `Progress` and `Stats` messages to be returned. You must provide a valid bucket name and an object that contains data in CSV format. For examples of how to use Amazon S3 Select with the AWS SDK for Java, see [Select content from an object](https://docs.aws.amazon.com/AmazonS3/latest/API/s3_example_s3_SelectObjectContent_section.html) in the *Amazon S3 API Reference*. ------ #### [ JavaScript ] For a JavaScript example that uses the AWS SDK for JavaScript with the S3 `SelectObjectContent` API operation to select records from JSON and CSV files that are stored in Amazon S3, see the blog post [ Introducing support for Amazon S3 Select in the AWS SDK for JavaScript](https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-javascript/). ------ #### [ Python ] For a Python example of using SQL queries to search through data that was loaded to Amazon S3 as a comma-separated value (CSV) file by using S3 Select, see the blog post [Querying data without servers or databases using Amazon S3 Select](https://aws.amazon.com/blogs/storage/querying-data-without-servers-or-databases-using-amazon-s3-select/). ------ # SQL reference for Amazon S3 Select **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) This reference contains a description of the structured query language (SQL) elements that are supported by Amazon S3 Select. **Topics** + [ # SELECT command ](s3-select-sql-reference-select.md) + [ # Data types ](s3-select-sql-reference-data-types.md) + [ # Operators ](s3-select-sql-reference-operators.md) + [ # Reserved keywords ](s3-select-sql-reference-keyword-list.md) + [ # SQL functions ](s3-select-sql-reference-sql-functions.md) # SELECT command **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports only the `SELECT` SQL command. The following ANSI standard clauses are supported for `SELECT`: + `SELECT` list + `FROM` clause + `WHERE` clause + `LIMIT` clause **Note** Amazon S3 Select queries currently do not support subqueries or joins. ## SELECT list The `SELECT` list names the columns, functions, and expressions that you want the query to return. The list represents the output of the query. ``` SELECT * SELECT projection1 AS column_alias_1, projection2 AS column_alias_2 ``` The first form of `SELECT` with the `*` (asterisk) returns every row that passed the `WHERE` clause, as-is. The second form of `SELECT` creates a row with user-defined output scalar expressions **`projection1`** and **`projection2`** for each column. ## FROM clause Amazon S3 Select supports the following forms of the `FROM` clause: ``` FROM table_name FROM table_name alias FROM table_name AS alias ``` In each form of the `FROM` clause, `table_name` is the `S3Object` that's being queried. Users coming from traditional relational databases can think of this as a database schema that contains multiple views over a table. Following standard SQL, the `FROM` clause creates rows that are filtered in the `WHERE` clause and projected in the `SELECT` list. For JSON objects that are stored in Amazon S3 Select, you can also use the following forms of the `FROM` clause: ``` FROM S3Object[*].path FROM S3Object[*].path alias FROM S3Object[*].path AS alias ``` Using this form of the `FROM` clause, you can select from arrays or objects within a JSON object. You can specify `path` by using one of the following forms: + By name (in an object): `.name` or `['name']` + By index (in an array): `[index]` + By wildcard character (in an object): `.*` + By wildcard character (in an array): `[*]` **Note** This form of the `FROM` clause works only with JSON objects. Wildcard characters always emit at least one record. If no record matches, then Amazon S3 Select emits the value `MISSING`. During output serialization (after the query finishes running), Amazon S3 Select replaces `MISSING` values with empty records. Aggregate functions (`AVG`, `COUNT`, `MAX`, `MIN`, and `SUM`) skip `MISSING` values. If you don't provide an alias when using a wildcard character, you can refer to the row by using the last element in the path. For example, you could select all prices from a list of books by using the query `SELECT price FROM S3Object[*].books[*].price`. If the path ends in a wildcard character instead of a name, then you can use the value `_1` to refer to the row. For example, instead of `SELECT price FROM S3Object[*].books[*].price`, you could use the query `SELECT _1.price FROM S3Object[*].books[*]`. Amazon S3 Select always treats a JSON document as an array of root-level values. Thus, even if the JSON object that you are querying has only one root element, the `FROM` clause must begin with `S3Object[*]`. However, for compatibility reasons, Amazon S3 Select allows you to omit the wildcard character if you don't include a path. Thus, the complete clause `FROM S3Object` is equivalent to `FROM S3Object[*] as S3Object`. If you include a path, you must also use the wildcard character. So, `FROM S3Object` and `FROM S3Object[*].path` are both valid clauses, but `FROM S3Object.path` is not. **Example** **Examples:** *Example \$11* This example shows results when using the following dataset and query: ``` { "Rules": [ {"id": "1"}, {"expr": "y > x"}, {"id": "2", "expr": "z = DEBUG"} ]} { "created": "June 27", "modified": "July 6" } ``` ``` SELECT id FROM S3Object[*].Rules[*].id ``` ``` {"id":"1"} {} {"id":"2"} {} ``` Amazon S3 Select produces each result for the following reasons: + `{"id":"id-1"}` – `S3Object[0].Rules[0].id` produced a match. + `{}` – `S3Object[0].Rules[1].id` did not match a record, so Amazon S3 Select emitted `MISSING`, which was then changed to an empty record during output serialization and returned. + `{"id":"id-2"}` – `S3Object[0].Rules[2].id` produced a match. + `{}` – `S3Object[1]` did not match on `Rules`, so Amazon S3 Select emitted `MISSING`, which was then changed to an empty record during output serialization and returned. If you don't want Amazon S3 Select to return empty records when it doesn't find a match, you can test for the value `MISSING`. The following query returns the same results as the previous query, but with the empty values omitted: ``` SELECT id FROM S3Object[*].Rules[*].id WHERE id IS NOT MISSING ``` ``` {"id":"1"} {"id":"2"} ``` *Example \$12* This example shows results when using the following dataset and queries: ``` { "created": "936864000", "dir_name": "important_docs", "files": [ { "name": "." }, { "name": ".." }, { "name": ".aws" }, { "name": "downloads" } ], "owner": "Amazon S3" } { "created": "936864000", "dir_name": "other_docs", "files": [ { "name": "." }, { "name": ".." }, { "name": "my stuff" }, { "name": "backup" } ], "owner": "User" } ``` ``` SELECT d.dir_name, d.files FROM S3Object[*] d ``` ``` {"dir_name":"important_docs","files":[{"name":"."},{"name":".."},{"name":".aws"},{"name":"downloads"}]} {"dir_name":"other_docs","files":[{"name":"."},{"name":".."},{"name":"my stuff"},{"name":"backup"}]} ``` ``` SELECT _1.dir_name, _1.owner FROM S3Object[*] ``` ``` {"dir_name":"important_docs","owner":"Amazon S3"} {"dir_name":"other_docs","owner":"User"} ``` ## WHERE clause The `WHERE` clause follows this syntax: ``` WHERE condition ``` The `WHERE` clause filters rows based on the `condition`. A condition is an expression that has a Boolean result. Only rows for which the condition evaluates to `TRUE` are returned in the result. ## LIMIT clause The `LIMIT` clause follows this syntax: ``` LIMIT number ``` The `LIMIT` clause limits the number of records that you want the query to return based on `number`. ## Attribute access The `SELECT` and `WHERE` clauses can refer to record data by using one of the methods in the following sections, depending on whether the file that is being queried is in CSV or JSON format. ### CSV + **Column Numbers** – You can refer to the *Nth* column of a row with the column name `_N`, where *`N`* is the column position. The position count starts at 1. For example, the first column is named `_1` and the second column is named `_2`. You can refer to a column as `_N` or `alias._N`. For example, `_2` and `myAlias._2` are both valid ways to refer to a column in the `SELECT` list and `WHERE` clause. + **Column Headers** – For objects in CSV format that have a header row, the headers are available to the `SELECT` list and `WHERE` clause. In particular, as in traditional SQL, within `SELECT` and `WHERE` clause expressions, you can refer to the columns by `alias.column_name` or `column_name`. ### JSON + **Document** – You can access JSON document fields as `alias.name`. You can also access nested fields, for example, `alias.name1.name2.name3`. + **List** – You can access elements in a JSON list by using zero-based indexes with the `[]` operator. For example, you can access the second element of a list as `alias[1]`. You can combine accessing list elements with fields, for example, `alias.name1.name2[1].name3`. + **Examples:** Consider this JSON object as a sample dataset: ``` {"name": "Susan Smith", "org": "engineering", "projects": [ {"project_name":"project1", "completed":false}, {"project_name":"project2", "completed":true} ] } ``` *Example \$11* The following query returns these results: ``` Select s.name from S3Object s ``` ``` {"name":"Susan Smith"} ``` *Example \$12* The following query returns these results: ``` Select s.projects[0].project_name from S3Object s ``` ``` {"project_name":"project1"} ``` ## Case sensitivity of header and attribute names With Amazon S3 Select, you can use double quotation marks to indicate that column headers (for CSV objects) and attributes (for JSON objects) are case sensitive. Without double quotation marks, object headers and attributes are case insensitive. An error is thrown in cases of ambiguity. The following examples are either 1) Amazon S3 objects in CSV format with the specified column headers, and with `FileHeaderInfo` set to `"Use"` for the query request; or 2) Amazon S3 objects in JSON format with the specified attributes. *Example \$11:* The object being queried has the header or attribute `NAME`. + The following expression successfully returns values from the object. Because there are no quotation marks, the query is case insensitive. ``` SELECT s.name from S3Object s ``` + The following expression results in a 400 error `MissingHeaderName`. Because there are quotation marks, the query is case sensitive. ``` SELECT s."name" from S3Object s ``` *Example \$12:* The Amazon S3 object being queried has one header or attribute with `NAME` and another header or attribute with `name`. + The following expression results in a 400 error `AmbiguousFieldName`. Because there are no quotation marks, the query is case insensitive, but there are two matches, so the error is thrown. ``` SELECT s.name from S3Object s ``` + The following expression successfully returns values from the object. Because there are quotation marks, the query is case sensitive, so there is no ambiguity. ``` SELECT s."NAME" from S3Object s ``` ## Using reserved keywords as user-defined terms Amazon S3 Select has a set of reserved keywords that are needed to run the SQL expressions used to query object content. Reserved keywords include function names, data types, operators, and so on. In some cases, user-defined terms, such as the column headers (for CSV files) or attributes (for JSON objects), might clash with a reserved keyword. When this happens, you must use double quotation marks to indicate that you are intentionally using a user-defined term that clashes with a reserved keyword. Otherwise a 400 parse error will result. For the full list of reserved keywords, see [Reserved keywords](s3-select-sql-reference-keyword-list.md). The following example is either 1) an Amazon S3 object in CSV format with the specified column headers, with `FileHeaderInfo` set to `"Use"` for the query request, or 2) an Amazon S3 object in JSON format with the specified attributes. *Example:* The object being queried has a header or attribute named `CAST`, which is a reserved keyword. + The following expression successfully returns values from the object. Because quotation marks are used in the query, S3 Select uses the user-defined header or attribute. ``` SELECT s."CAST" from S3Object s ``` + The following expression results in a 400 parse error. Because no quotation marks are used in the query, `CAST` clashes with a reserved keyword. ``` SELECT s.CAST from S3Object s ``` ## Scalar expressions Within the `WHERE` clause and the `SELECT` list, you can have SQL *scalar expressions*, which are expressions that return scalar values. They have the following form: + ***`literal`*** An SQL literal. + ***`column_reference`*** A reference to a column in the form `column_name` or `alias.column_name`. + **`unary_op`** **`expression`** In this case, ****`unary_op`**** is an SQL unary operator. + **`expression`** **`binary_op`** ***`expression`*** In this case, ****`binary_op`**** is an SQL binary operator. + **`func_name`** In this case, **`func_name`** is the name of the scalar function to invoke. + ***`expression`*** `[ NOT ] BETWEEN` ****`expression`**** `AND` ****`expression`**** + ***`expression`*** `LIKE` ****`expression`**** [ `ESCAPE` ***`expression`*** ] # Data types **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports several primitive data types. ## Data type conversions The general rule is to follow the `CAST` function if it's defined. If `CAST` is not defined, then all input data is treated as a string. In that case, you must cast your input data into the relevant data types when necessary. For more information about the `CAST` function, see [CAST](s3-select-sql-reference-conversion.md#s3-select-sql-reference-cast). ## Supported data types Amazon S3 Select supports the following set of primitive data types. | Name | Description | Examples | | --- | --- | --- | | `bool` | A Boolean value, either `TRUE` or `FALSE`. | `FALSE` | | `int`, `integer` | An 8-byte signed integer in the range -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807. | `100000` | | `string` | A UTF8-encoded variable-length string. The default limit is 1 character. The maximum character limit is 2,147,483,647. | `'xyz'` | | `float` | An 8-byte floating point number. | `CAST(0.456 AS FLOAT)` | | `decimal`, `numeric` | A base-10 number, with a maximum precision of 38 (that is, the maximum number of significant digits), and with a scale within the range of -231 to 231-1 (that is, the base-10 exponent). Amazon S3 Select ignores scale and precision when you provide both at the same time. | `123.456 ` | | `timestamp` | Timestamps represent a specific moment in time, always include a local offset, and are capable of arbitrary precision. In the text format, timestamps follow the [W3C note on date and time formats](https://www.w3.org/TR/NOTE-datetime), but they must end with the literal `T` if the timestamps are not at least whole-day precision. Fractional seconds are allowed, with at least one digit of precision, and an unlimited maximum. Local-time offsets can be represented as either hour:minute offsets from UTC, or as the literal `Z` to denote a local time of UTC. Local-time offsets are required on timestamps with time and are not allowed on date values. | `CAST('2007-04-05T14:30Z' AS TIMESTAMP)` | ### Supported Parquet types Amazon S3 Select supports the following Parquet types. + `DATE` + `DECIMAL` + `ENUM` + `INT(8)` + `INT(16)` + `INT(32)` + `INT(64)` + `LIST` **Note** For `LIST` Parquet type output, Amazon S3 Select supports only JSON format. However, if the query limits the data to simple values, the `LIST` Parquet type can also be queried in CSV format. + `STRING` + `TIMESTAMP` supported precision (`MILLIS`/`MICROS`/`NANOS`) **Note** Timestamps saved as an `INT(96)` are unsupported. Because of the range of the `INT(64)` type, timestamps that are using the `NANOS` unit can represent only values between `1677-09-21 00:12:43` and `2262-04-11 23:47:16`. Values outside of this range cannot be represented with the `NANOS` unit. ### Mapping of Parquet types to supported data types in Amazon S3 Select | Parquet types | Supported data types | | --- | --- | | `DATE` | `timestamp` | | `DECIMAL` | `decimal`, `numeric` | | `ENUM` | `string` | | `INT(8)` | `int`, `integer` | | `INT(16)` | `int`, `integer` | | `INT(32)` | `int`, `integer` | | `INT(64)` | `decimal`, `numeric` | | `LIST` | Each Parquet type in list is mapped to the corresponding data type. | | `STRING` | `string` | | `TIMESTAMP` | `timestamp` | # Operators **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports the following operators. ## Logical operators + `AND` + `NOT` + `OR` ## Comparison operators + `<` + `>` + `<=` + `>=` + `=` + `<>` + `!=` + `BETWEEN` + `IN` – For example: `IN ('a', 'b', 'c')` ## Pattern-matching operators + `LIKE` + `_` (Matches any character) + `%` (Matches any sequence of characters) ## Unitary operators + `IS NULL` + `IS NOT NULL` ## Math operators Addition, subtraction, multiplication, division, and modulo are supported, as follows: + \$1 + - + \$1 + / + % ## Operator precedence The following table shows the operators' precedence in decreasing order. | Operator or element | Associativity | Required | | --- | --- | --- | | `-` | right | unary minus | | `*`, `/`, `%` | left | multiplication, division, modulo | | `+`, `-` | left | addition, subtraction | | `IN` | | set membership | | `BETWEEN` | | range containment | | `LIKE` | | string pattern matching | | `<``>` | | less than, greater than | | `=` | right | equality, assignment | | `NOT` | right | logical negation | | `AND` | left | logical conjunction | | `OR` | left | logical disjunction | # Reserved keywords **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) The following is the list of reserved keywords for Amazon S3 Select. These keywords include function names, data types, operators, and so on, that are needed to run the SQL expressions that are used to query object content. ``` absolute action add all allocate alter and any are as asc assertion at authorization avg bag begin between bit bit_length blob bool boolean both by cascade cascaded case cast catalog char char_length character character_length check clob close coalesce collate collation column commit connect connection constraint constraints continue convert corresponding count create cross current current_date current_time current_timestamp current_user cursor date day deallocate dec decimal declare default deferrable deferred delete desc describe descriptor diagnostics disconnect distinct domain double drop else end end-exec escape except exception exec execute exists external extract false fetch first float for foreign found from full get global go goto grant group having hour identity immediate in indicator initially inner input insensitive insert int integer intersect interval into is isolation join key language last leading left level like limit list local lower match max min minute missing module month names national natural nchar next no not null nullif numeric octet_length of on only open option or order outer output overlaps pad partial pivot position precision prepare preserve primary prior privileges procedure public read real references relative restrict revoke right rollback rows schema scroll second section select session session_user set sexp size smallint some space sql sqlcode sqlerror sqlstate string struct substring sum symbol system_user table temporary then time timestamp timezone_hour timezone_minute to trailing transaction translate translation trim true tuple union unique unknown unpivot update upper usage user using value values varchar varying view when whenever where with work write year zone ``` # SQL functions **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports the following SQL functions. **Topics** + [ # Aggregate functions ](s3-select-sql-reference-aggregate.md) + [ # Conditional functions ](s3-select-sql-reference-conditional.md) + [ # Conversion functions ](s3-select-sql-reference-conversion.md) + [ # Date functions ](s3-select-sql-reference-date.md) + [ # String functions ](s3-select-sql-reference-string.md) # Aggregate functions **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports the following aggregate functions. | Function | Argument type | Return type | | --- | --- | --- | | `AVG(expression)` | `INT`, `FLOAT`, `DECIMAL` | `DECIMAL` for an `INT` argument, `FLOAT` for a floating-point argument; otherwise the same as the argument data type. | | `COUNT` | `-` | `INT` | | `MAX(expression)` | `INT`, `DECIMAL` | Same as the argument type. | | `MIN(expression)` | `INT`, `DECIMAL` | Same as the argument type. | | `SUM(expression)` | `INT`, `FLOAT`, `DOUBLE`, `DECIMAL` | `INT` for an `INT` argument, `FLOAT` for a floating-point argument; otherwise, the same as the argument data type. | ## SUM example To aggregate the total object sizes of a folder in an [S3 Inventory report](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html), use a `SUM` expression. The following S3 Inventory report is a CSV file that's compressed with GZIP. There are three columns. + The first column is the name of the S3 bucket (*`DOC-EXAMPLE-BUCKET`*) that the S3 Inventory report is for. + The second column is the object key name that uniquely identifies the object in the bucket. The `example-folder/` value in the first row is for the folder `example-folder`. In Amazon S3, when you create a folder in your bucket, S3 creates a 0-byte object with a key that's set to the folder name that you provided. The `example-folder/object1` value in the second row is for the object `object1` in the folder `example-folder`. The `example-folder/object2` value in the third row is for the object `object2` in the folder `example-folder`. For more information about S3 folders, see [Organizing objects in the Amazon S3 console by using folders](using-folders.md). + The third column is the object size in bytes. ``` "DOC-EXAMPLE-BUCKET","example-folder/","0" "DOC-EXAMPLE-BUCKET","example-folder/object1","2011267" "DOC-EXAMPLE-BUCKET","example-folder/object2","1570024" ``` To use a `SUM` expression to calculate the total size of the folder `example-folder`, run the SQL query with Amazon S3 Select. ``` SELECT SUM(CAST(_3 as INT)) FROM s3object s WHERE _2 LIKE 'example-folder/%' AND _2 != 'example-folder/'; ``` Query Result: ``` 3581291 ``` # Conditional functions **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports the following conditional functions. **Topics** + [ ## CASE ](#s3-select-sql-reference-case) + [ ## COALESCE ](#s3-select-sql-reference-coalesce) + [ ## NULLIF ](#s3-select-sql-reference-nullif) ## CASE The `CASE` expression is a conditional expression, similar to `if/then/else` statements found in other languages. `CASE` is used to specify a result when there are multiple conditions. There are two types of `CASE` expressions: simple and searched. In simple `CASE` expressions, an expression is compared with a value. When a match is found, the specified action in the `THEN` clause is applied. If no match is found, the action in the `ELSE` clause is applied. In searched `CASE` expressions, each `CASE` is evaluated based on a Boolean expression, and the `CASE` statement returns the first matching `CASE`. If no matching `CASE` is found among the `WHEN` clauses, the action in the `ELSE` clause is returned. ### Syntax **Note** Currently, Amazon S3 Select doesn't support `ORDER BY` or queries that contain new lines. Make sure that you use queries with no line breaks. The following is a simple `CASE` statement that's used to match conditions: ``` CASE expression WHEN value THEN result [WHEN...] [ELSE result] END ``` The following is a searched `CASE` statement that's used to evaluate each condition: ``` CASE WHEN boolean condition THEN result [WHEN ...] [ELSE result] END ``` ### Examples **Note** If you use the Amazon S3 console to run the following examples and your CSV file contains a header row, choose **Exclude the first line of CSV data**. **Example 1:** Use a simple `CASE` expression to replace `New York City` with `Big Apple` in a query. Replace all other city names with `other`. ``` SELECT venuecity, CASE venuecity WHEN 'New York City' THEN 'Big Apple' ELSE 'other' END FROM S3Object; ``` Query result: ``` venuecity | case -----------------+----------- Los Angeles | other New York City | Big Apple San Francisco | other Baltimore | other ... ``` **Example 2:** Use a searched `CASE` expression to assign group numbers based on the `pricepaid` value for individual ticket sales: ``` SELECT pricepaid, CASE WHEN CAST(pricepaid as FLOAT) < 10000 THEN 'group 1' WHEN CAST(pricepaid as FLOAT) > 10000 THEN 'group 2' ELSE 'group 3' END FROM S3Object; ``` Query result: ``` pricepaid | case -----------+--------- 12624.00 | group 2 10000.00 | group 3 10000.00 | group 3 9996.00 | group 1 9988.00 | group 1 ... ``` ## COALESCE `COALESCE` evaluates the arguments in order and returns the first non-unknown value, that is, the first non-null or non-missing value. This function does not propagate null and missing values. ### Syntax ``` COALESCE ( expression, expression, ... ) ``` ### Parameters *`expression`* The target expression that the function operates on. ### Examples ``` COALESCE(1) -- 1 COALESCE(null) -- null COALESCE(null, null) -- null COALESCE(missing) -- null COALESCE(missing, missing) -- null COALESCE(1, null) -- 1 COALESCE(null, null, 1) -- 1 COALESCE(null, 'string') -- 'string' COALESCE(missing, 1) -- 1 ``` ## NULLIF Given two expressions, `NULLIF` returns `NULL` if the two expressions evaluate to the same value; otherwise, `NULLIF` returns the result of evaluating the first expression. ### Syntax ``` NULLIF ( expression1, expression2 ) ``` ### Parameters `expression1, expression2` The target expressions that the function operates on. ### Examples ``` NULLIF(1, 1) -- null NULLIF(1, 2) -- 1 NULLIF(1.0, 1) -- null NULLIF(1, '1') -- 1 NULLIF([1], [1]) -- null NULLIF(1, NULL) -- 1 NULLIF(NULL, 1) -- null NULLIF(null, null) -- null NULLIF(missing, null) -- null NULLIF(missing, missing) -- null ``` # Conversion functions **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports the following conversion function. **Topics** + [ ## CAST ](#s3-select-sql-reference-cast) ## CAST The `CAST` function converts an entity, such as an expression that evaluates to a single value, from one type to another. ### Syntax ``` CAST ( expression AS data_type ) ``` ### Parameters *`expression`* A combination of one or more values, operators, and SQL functions that evaluate to a value. *`data_type`* The target data type, such as `INT`, to cast the expression to. For a list of supported data types, see [Data types](s3-select-sql-reference-data-types.md). ### Examples ``` CAST('2007-04-05T14:30Z' AS TIMESTAMP) CAST(0.456 AS FLOAT) ``` # Date functions **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports the following date functions. **Topics** + [ ## DATE\$1ADD ](#s3-select-sql-reference-date-add) + [ ## DATE\$1DIFF ](#s3-select-sql-reference-date-diff) + [ ## EXTRACT ](#s3-select-sql-reference-extract) + [ ## TO\$1STRING ](#s3-select-sql-reference-to-string) + [ ## TO\$1TIMESTAMP ](#s3-select-sql-reference-to-timestamp) + [ ## UTCNOW ](#s3-select-sql-reference-utcnow) ## DATE\$1ADD Given a date part, a quantity, and a timestamp, `DATE_ADD` returns an updated timestamp by altering the date part by the quantity. ### Syntax ``` DATE_ADD( date_part, quantity, timestamp ) ``` ### Parameters *`date_part`* Specifies which part of the date to modify. This can be one of the following: + year + month + day + hour + minute + second *`quantity`* The value to apply to the updated timestamp. Positive values for `quantity` add to the timestamp's date\$1part, and negative values subtract. *`timestamp`* The target timestamp that the function operates on. ### Examples ``` DATE_ADD(year, 5, `2010-01-01T`) -- 2015-01-01 (equivalent to 2015-01-01T) DATE_ADD(month, 1, `2010T`) -- 2010-02T (result will add precision as necessary) DATE_ADD(month, 13, `2010T`) -- 2011-02T DATE_ADD(day, -1, `2017-01-10T`) -- 2017-01-09 (equivalent to 2017-01-09T) DATE_ADD(hour, 1, `2017T`) -- 2017-01-01T01:00-00:00 DATE_ADD(hour, 1, `2017-01-02T03:04Z`) -- 2017-01-02T04:04Z DATE_ADD(minute, 1, `2017-01-02T03:04:05.006Z`) -- 2017-01-02T03:05:05.006Z DATE_ADD(second, 1, `2017-01-02T03:04:05.006Z`) -- 2017-01-02T03:04:06.006Z ``` ## DATE\$1DIFF Given a date part and two valid timestamps, `DATE_DIFF` returns the difference in date parts. The return value is a negative integer when the `date_part` value of `timestamp1` is greater than the `date_part` value of `timestamp2`. The return value is a positive integer when the `date_part` value of `timestamp1` is less than the `date_part` value of `timestamp2`. ### Syntax ``` DATE_DIFF( date_part, timestamp1, timestamp2 ) ``` ### Parameters **`date_part`** Specifies which part of the timestamps to compare. For the definition of `date_part`, see [DATE\$1ADD](#s3-select-sql-reference-date-add). **`timestamp1`** The first timestamp to compare. **`timestamp2`** The second timestamp to compare. ### Examples ``` DATE_DIFF(year, `2010-01-01T`, `2011-01-01T`) -- 1 DATE_DIFF(year, `2010T`, `2010-05T`) -- 4 (2010T is equivalent to 2010-01-01T00:00:00.000Z) DATE_DIFF(month, `2010T`, `2011T`) -- 12 DATE_DIFF(month, `2011T`, `2010T`) -- -12 DATE_DIFF(day, `2010-01-01T23:00`, `2010-01-02T01:00`) -- 0 (need to be at least 24h apart to be 1 day apart) ``` ## EXTRACT Given a date part and a timestamp, `EXTRACT` returns the timestamp's date part value. ### Syntax ``` EXTRACT( date_part FROM timestamp ) ``` ### Parameters **`date_part`** Specifies which part of the timestamps to extract. This can be one of the following: + `YEAR` + `MONTH` + `DAY` + `HOUR` + `MINUTE` + `SECOND` + `TIMEZONE_HOUR` + `TIMEZONE_MINUTE` **`timestamp`** The target timestamp that the function operates on. ### Examples ``` EXTRACT(YEAR FROM `2010-01-01T`) -- 2010 EXTRACT(MONTH FROM `2010T`) -- 1 (equivalent to 2010-01-01T00:00:00.000Z) EXTRACT(MONTH FROM `2010-10T`) -- 10 EXTRACT(HOUR FROM `2017-01-02T03:04:05+07:08`) -- 3 EXTRACT(MINUTE FROM `2017-01-02T03:04:05+07:08`) -- 4 EXTRACT(TIMEZONE_HOUR FROM `2017-01-02T03:04:05+07:08`) -- 7 EXTRACT(TIMEZONE_MINUTE FROM `2017-01-02T03:04:05+07:08`) -- 8 ``` ## TO\$1STRING Given a timestamp and a format pattern, `TO_STRING` returns a string representation of the timestamp in the given format. ### Syntax ``` TO_STRING ( timestamp time_format_pattern ) ``` ### Parameters *`timestamp`* The target timestamp that the function operates on. *`time_format_pattern`* A string that has the following special character interpretations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-select-sql-reference-date.html) ### Examples ``` TO_STRING(`1969-07-20T20:18Z`, 'MMMM d, y') -- "July 20, 1969" TO_STRING(`1969-07-20T20:18Z`, 'MMM d, yyyy') -- "Jul 20, 1969" TO_STRING(`1969-07-20T20:18Z`, 'M-d-yy') -- "7-20-69" TO_STRING(`1969-07-20T20:18Z`, 'MM-d-y') -- "07-20-1969" TO_STRING(`1969-07-20T20:18Z`, 'MMMM d, y h:m a') -- "July 20, 1969 8:18 PM" TO_STRING(`1969-07-20T20:18Z`, 'y-MM-dd''T''H:m:ssX') -- "1969-07-20T20:18:00Z" TO_STRING(`1969-07-20T20:18+08:00Z`, 'y-MM-dd''T''H:m:ssX') -- "1969-07-20T20:18:00Z" TO_STRING(`1969-07-20T20:18+08:00`, 'y-MM-dd''T''H:m:ssXXXX') -- "1969-07-20T20:18:00+0800" TO_STRING(`1969-07-20T20:18+08:00`, 'y-MM-dd''T''H:m:ssXXXXX') -- "1969-07-20T20:18:00+08:00" ``` ## TO\$1TIMESTAMP Given a string, `TO_TIMESTAMP` converts it to a timestamp. `TO_TIMESTAMP` is the inverse operation of `TO_STRING`. ### Syntax ``` TO_TIMESTAMP ( string ) ``` ### Parameters *`string`* The target string that the function operates on. ### Examples ``` TO_TIMESTAMP('2007T') -- `2007T` TO_TIMESTAMP('2007-02-23T12:14:33.079-08:00') -- `2007-02-23T12:14:33.079-08:00` ``` ## UTCNOW `UTCNOW` returns the current time in UTC as a timestamp. ### Syntax ``` UTCNOW() ``` ### Parameters `UTCNOW` takes no parameters. ### Examples ``` UTCNOW() -- 2017-10-13T16:02:11.123Z ``` # String functions **Important** Amazon S3 Select is no longer available to new customers. Existing customers of Amazon S3 Select can continue to use the feature as usual. [Learn more](https://aws.amazon.com/blogs/storage/how-to-optimize-querying-your-data-in-amazon-s3/) Amazon S3 Select supports the following string functions. **Topics** + [ ## CHAR\$1LENGTH, CHARACTER\$1LENGTH ](#s3-select-sql-reference-char-length) + [ ## LOWER ](#s3-select-sql-reference-lower) + [ ## SUBSTRING ](#s3-select-sql-reference-substring) + [ ## TRIM ](#s3-select-sql-reference-trim) + [ ## UPPER ](#s3-select-sql-reference-upper) ## CHAR\$1LENGTH, CHARACTER\$1LENGTH `CHAR_LENGTH` (or `CHARACTER_LENGTH`) counts the number of characters in the specified string. **Note** `CHAR_LENGTH` and `CHARACTER_LENGTH` are synonyms. ### Syntax ``` CHAR_LENGTH ( string ) ``` ### Parameters *`string`* The target string that the function operates on. ### Examples ``` CHAR_LENGTH('') -- 0 CHAR_LENGTH('abcdefg') -- 7 ``` ## LOWER Given a string, `LOWER` converts all uppercase characters to lowercase characters. Any non-uppercased characters remain unchanged. ### Syntax ``` LOWER ( string ) ``` ### Parameters **`string`** The target string that the function operates on. ### Examples ``` LOWER('AbCdEfG!@#$') -- 'abcdefg!@#$' ``` ## SUBSTRING Given a string, a start index, and optionally a length, `SUBSTRING` returns the substring from the start index up to the end of the string, or up to the length provided. **Note** The first character of the input string has an index position of 1. If `start` is < 1, with no length specified, then the index position is set to 1. If `start` is < 1, with a length specified, then the index position is set to `start + length -1`. If `start + length -1` < 0, then an empty string is returned. If `start + length -1` > = 0, then the substring starting at index position 1 with the length `start + length - 1` is returned. ### Syntax ``` SUBSTRING( string FROM start [ FOR length ] ) ``` ### Parameters **`string`** The target string that the function operates on. **`start`** The start position of the string. **`length`** The length of the substring to return. If not present, proceed to the end of the string. ### Examples ``` SUBSTRING("123456789", 0) -- "123456789" SUBSTRING("123456789", 1) -- "123456789" SUBSTRING("123456789", 2) -- "23456789" SUBSTRING("123456789", -4) -- "123456789" SUBSTRING("123456789", 0, 999) -- "123456789" SUBSTRING("123456789", 1, 5) -- "12345" ``` ## TRIM Trims leading or trailing characters from a string. The default character to remove is a space (`' '`). ### Syntax ``` TRIM ( [[LEADING | TRAILING | BOTH remove_chars] FROM] string ) ``` ### Parameters **`string`** The target string that the function operates on. `LEADING` \$1 `TRAILING` \$1 `BOTH` This parameter indicates whether to trim leading or trailing characters, or both leading and trailing characters. **`remove_chars`** The set of characters to remove. `remove_chars` can be a string with a length > 1. This function returns the string with any character from `remove_chars` found at the beginning or end of the string that was removed. ### Examples ``` TRIM(' foobar ') -- 'foobar' TRIM(' \tfoobar\t ') -- '\tfoobar\t' TRIM(LEADING FROM ' foobar ') -- 'foobar ' TRIM(TRAILING FROM ' foobar ') -- ' foobar' TRIM(BOTH FROM ' foobar ') -- 'foobar' TRIM(BOTH '12' FROM '1112211foobar22211122') -- 'foobar' ``` ## UPPER Given a string, `UPPER` converts all lowercase characters to uppercase characters. Any non-lowercased characters remain unchanged. ### Syntax ``` UPPER ( string ) ``` ### Parameters **`string`** The target string that the function operates on. ### Examples ``` UPPER('AbCdEfG!@#$') -- 'ABCDEFG!@#$' ```