# Working with S3 on Outposts objects With Amazon S3 on Outposts, you can create S3 buckets on your AWS Outposts and easily store and retrieve objects on premises for applications that require local data access, local data processing, and data residency. S3 on Outposts provides a new storage class, S3 Outposts (`OUTPOSTS`), which uses the Amazon S3 APIs, and is designed to store data durably and redundantly across multiple devices and servers on your AWS Outposts. You communicate with your Outpost bucket by using an access point and endpoint connection over a virtual private cloud (VPC). You can use the same APIs and features on Outpost buckets as you do on Amazon S3 buckets, including access policies, encryption, and tagging. You can use S3 on Outposts through the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS SDKs, or REST API. Objects are the fundamental entities stored in Amazon S3 on Outposts. Every object is contained in a bucket. You must use access points to access any object in an Outpost bucket. When you specify the bucket for object operations, you use the access point Amazon Resource Name (ARN) or the access point alias. For more information about access point aliases, see [Using a bucket-style alias for your S3 on Outposts bucket access point](s3-outposts-access-points-alias.md). The following example shows the ARN format for S3 on Outposts access points, which includes the AWS Region code for the Region that the Outpost is homed to, the AWS account ID, the Outpost ID, and the access point name: ``` arn:aws:s3-outposts:region:account-id:outpost/outpost-id/accesspoint/accesspoint-name ``` For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). Object ARNs use the following format, which includes the AWS Region that the Outpost is homed to, AWS account ID, Outpost ID, bucket name, and object key: ``` arn:aws:s3-outposts:us-west-2:123456789012:​outpost/op-01ac5d28a6a232904/bucket/amzn-s3-demo-bucket1/object/myobject ``` With Amazon S3 on Outposts, object data is always stored on the Outpost. When AWS installs an Outpost rack, your data stays local to your Outpost to meet data-residency requirements. Your objects never leave your Outpost and are not in an AWS Region. Because the AWS Management Console is hosted in-Region, you can't use the console to upload or manage objects in your Outpost. However, you can use the REST API, AWS Command Line Interface (AWS CLI), and AWS SDKs to upload and manage your objects through your access points. **Topics** + [ # Upload an object to an S3 on Outposts bucket ](S3OutpostsUploadObjects.md) + [ # Copying an object in an Amazon S3 on Outposts bucket using the AWS SDK for Java ](S3OutpostsCopyObject.md) + [ # Getting an object from an Amazon S3 on Outposts bucket ](S3OutpostsGetObject.md) + [ # Listing the objects in an Amazon S3 on Outposts bucket ](S3OutpostsListObjects.md) + [ # Deleting objects in Amazon S3 on Outposts buckets ](S3OutpostsDeleteObject.md) + [ # Using HeadBucket to determine if an S3 on Outposts bucket exists and you have access permissions ](S3OutpostsHeadBucket.md) + [ # Performing and managing a multipart upload with the SDK for Java ](S3OutpostsMPU.md) + [ # Using presigned URLs for S3 on Outposts ](S3OutpostsPresignedURL.md) + [ # Amazon S3 on Outposts with local Amazon EMR on Outposts ](s3-outposts-emr.md) + [ # Authorization and authentication caching ](s3-outposts-auth-cache.md) # Upload an object to an S3 on Outposts bucket Objects are the fundamental entities stored in Amazon S3 on Outposts. Every object is contained in a bucket. You must use access points to access any object in an Outpost bucket. When you specify the bucket for object operations, you use the access point Amazon Resource Name (ARN) or the access point alias. For more information about access point aliases, see [Using a bucket-style alias for your S3 on Outposts bucket access point](s3-outposts-access-points-alias.md). The following example shows the ARN format for S3 on Outposts access points, which includes the AWS Region code for the Region that the Outpost is homed to, the AWS account ID, the Outpost ID, and the access point name: ``` arn:aws:s3-outposts:region:account-id:outpost/outpost-id/accesspoint/accesspoint-name ``` For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). With Amazon S3 on Outposts, object data is always stored on the Outpost. When AWS installs an Outpost rack, your data stays local to your Outpost to meet data-residency requirements. Your objects never leave your Outpost and are not in an AWS Region. Because the AWS Management Console is hosted in-Region, you can't use the console to upload or manage objects in your Outpost. However, you can use the REST API, AWS Command Line Interface (AWS CLI), and AWS SDKs to upload and manage your objects through your access points. The following AWS CLI and AWS SDK for Java examples show you how to upload an object to an S3 on Outposts bucket by using an access point. ------ #### [ AWS CLI ] **Example** The following example puts an object named `sample-object.xml` into an S3 on Outposts bucket (`s3-outposts:PutObject`) by using the AWS CLI. To use this command, replace each `user input placeholder` with your own information. For more information about this command, see [put-object](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-object.html) in the *AWS CLI Reference*. ``` aws s3api put-object --bucket arn:aws:s3-outposts:Region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-outposts-access-point --key sample-object.xml --body sample-object.xml ``` ------ #### [ SDK for Java ] **Example** For examples of how to upload an object to an S3 Outposts bucket with the AWS SDK for Java, see [PutObjectOnOutpost.java](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/s3/src/main/java/com/example/s3/outposts/PutObjectOnOutpost.java) in the *AWS SDK for Java 2.x Code Examples*. ------ # Copying an object in an Amazon S3 on Outposts bucket using the AWS SDK for Java Objects are the fundamental entities stored in Amazon S3 on Outposts. Every object is contained in a bucket. You must use access points to access any object in an Outpost bucket. When you specify the bucket for object operations, you use the access point Amazon Resource Name (ARN) or the access point alias. For more information about access point aliases, see [Using a bucket-style alias for your S3 on Outposts bucket access point](s3-outposts-access-points-alias.md). The following example shows the ARN format for S3 on Outposts access points, which includes the AWS Region code for the Region that the Outpost is homed to, the AWS account ID, the Outpost ID, and the access point name: ``` arn:aws:s3-outposts:region:account-id:outpost/outpost-id/accesspoint/accesspoint-name ``` For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). With Amazon S3 on Outposts, object data is always stored on the Outpost. When AWS installs an Outpost rack, your data stays local to your Outpost to meet data-residency requirements. Your objects never leave your Outpost and are not in an AWS Region. Because the AWS Management Console is hosted in-Region, you can't use the console to upload or manage objects in your Outpost. However, you can use the REST API, AWS Command Line Interface (AWS CLI), and AWS SDKs to upload and manage your objects through your access points. The following example shows you how to copy an object in an S3 on Outposts bucket by using the AWS SDK for Java. ## Using the AWS SDK for Java The following S3 on Outposts example copies an object into a new object in the same bucket by using the SDK for Java. To use this example, replace the `user input placeholders` with your own information. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.CopyObjectRequest; public class CopyObject { public static void main(String[] args) { String accessPointArn = "*** access point ARN ***"; String sourceKey = "*** Source object key ***"; String destinationKey = "*** Destination object key ***"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); // Copy the object into a new object in the same bucket. CopyObjectRequest copyObjectRequest = new CopyObjectRequest(accessPointArn, sourceKey, accessPointArn, destinationKey); s3Client.copyObject(copyObjectRequest); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` # Getting an object from an Amazon S3 on Outposts bucket Objects are the fundamental entities stored in Amazon S3 on Outposts. Every object is contained in a bucket. You must use access points to access any object in an Outpost bucket. When you specify the bucket for object operations, you use the access point Amazon Resource Name (ARN) or the access point alias. For more information about access point aliases, see [Using a bucket-style alias for your S3 on Outposts bucket access point](s3-outposts-access-points-alias.md). The following example shows the ARN format for S3 on Outposts access points, which includes the AWS Region code for the Region that the Outpost is homed to, the AWS account ID, the Outpost ID, and the access point name: ``` arn:aws:s3-outposts:region:account-id:outpost/outpost-id/accesspoint/accesspoint-name ``` For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). With Amazon S3 on Outposts, object data is always stored on the Outpost. When AWS installs an Outpost rack, your data stays local to your Outpost to meet data-residency requirements. Your objects never leave your Outpost and are not in an AWS Region. Because the AWS Management Console is hosted in-Region, you can't use the console to upload or manage objects in your Outpost. However, you can use the REST API, AWS Command Line Interface (AWS CLI), and AWS SDKs to upload and manage your objects through your access points. The following examples show you how to download (get) an object by using the AWS Command Line Interface (AWS CLI) and AWS SDK for Java. ## Using the AWS CLI The following example gets an object named `sample-object.xml` from an S3 on Outposts bucket (`s3-outposts:GetObject`) by using the AWS CLI. To use this command, replace each `user input placeholder` with your own information. For more information about this command, see [get-object](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/get-object.html) in the *AWS CLI Reference*. ``` aws s3api get-object --bucket arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-outposts-access-point --key testkey sample-object.xml ``` ## Using the AWS SDK for Java The following S3 on Outposts example gets an object by using the SDK for Java. To use this example, replace each `user input placeholder` with your own information. For more information, see [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) in the *Amazon Simple Storage Service API Reference*. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.GetObjectRequest; import com.amazonaws.services.s3.model.ResponseHeaderOverrides; import com.amazonaws.services.s3.model.S3Object; import java.io.BufferedReader; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; public class GetObject { public static void main(String[] args) throws IOException { String accessPointArn = "*** access point ARN ***"; String key = "*** Object key ***"; S3Object fullObject = null, objectPortion = null, headerOverrideObject = null; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); // Get an object and print its contents. System.out.println("Downloading an object"); fullObject = s3Client.getObject(new GetObjectRequest(accessPointArn, key)); System.out.println("Content-Type: " + fullObject.getObjectMetadata().getContentType()); System.out.println("Content: "); displayTextInputStream(fullObject.getObjectContent()); // Get a range of bytes from an object and print the bytes. GetObjectRequest rangeObjectRequest = new GetObjectRequest(accessPointArn, key) .withRange(0, 9); objectPortion = s3Client.getObject(rangeObjectRequest); System.out.println("Printing bytes retrieved."); displayTextInputStream(objectPortion.getObjectContent()); // Get an entire object, overriding the specified response headers, and print the object's content. ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides() .withCacheControl("No-cache") .withContentDisposition("attachment; filename=example.txt"); GetObjectRequest getObjectRequestHeaderOverride = new GetObjectRequest(accessPointArn, key) .withResponseHeaders(headerOverrides); headerOverrideObject = s3Client.getObject(getObjectRequestHeaderOverride); displayTextInputStream(headerOverrideObject.getObjectContent()); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } finally { // To ensure that the network connection doesn't remain open, close any open input streams. if (fullObject != null) { fullObject.close(); } if (objectPortion != null) { objectPortion.close(); } if (headerOverrideObject != null) { headerOverrideObject.close(); } } } private static void displayTextInputStream(InputStream input) throws IOException { // Read the text input stream one line at a time and display each line. BufferedReader reader = new BufferedReader(new InputStreamReader(input)); String line = null; while ((line = reader.readLine()) != null) { System.out.println(line); } System.out.println(); } } ``` # Listing the objects in an Amazon S3 on Outposts bucket Objects are the fundamental entities stored in Amazon S3 on Outposts. Every object is contained in a bucket. You must use access points to access any object in an Outpost bucket. When you specify the bucket for object operations, you use the access point Amazon Resource Name (ARN) or the access point alias. For more information about access point aliases, see [Using a bucket-style alias for your S3 on Outposts bucket access point](s3-outposts-access-points-alias.md). The following example shows the ARN format for S3 on Outposts access points, which includes the AWS Region code for the Region that the Outpost is homed to, the AWS account ID, the Outpost ID, and the access point name: ``` arn:aws:s3-outposts:region:account-id:outpost/outpost-id/accesspoint/accesspoint-name ``` For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). **Note** With Amazon S3 on Outposts, object data is always stored on the Outpost. When AWS installs an Outpost rack, your data stays local to your Outpost to meet data-residency requirements. Your objects never leave your Outpost and are not in an AWS Region. Because the AWS Management Console is hosted in-Region, you can't use the console to upload or manage objects in your Outpost. However, you can use the REST API, AWS Command Line Interface (AWS CLI), and AWS SDKs to upload and manage your objects through your access points. The following examples show you how to list the objects in an S3 on Outposts bucket using the AWS CLI and AWS SDK for Java. ## Using the AWS CLI The following example lists the objects in an S3 on Outposts bucket (`s3-outposts:ListObjectsV2`) by using the AWS CLI. To use this command, replace each `user input placeholder` with your own information. For more information about this command, see [list-objects-v2](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/list-objects-v2.html) in the *AWS CLI Reference*. ``` aws s3api list-objects-v2 --bucket arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-outposts-access-point ``` **Note** When using this action with Amazon S3 on Outposts through the AWS SDKs, you provide the Outposts access point ARN in place of the bucket name, in the following form: `arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-Outposts-Access-Point`. For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). ## Using the AWS SDK for Java The following S3 on Outposts example lists objects in a bucket by using the SDK for Java. To use this example, replace each `user input placeholder` with your own information. **Important** This example uses [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html), which is the latest revision of the `ListObjects` API operation. We recommend that you use this revised API operation for application development. For backward compatibility, Amazon S3 continues to support the prior version of this API operation. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.ListObjectsV2Request; import com.amazonaws.services.s3.model.ListObjectsV2Result; import com.amazonaws.services.s3.model.S3ObjectSummary; public class ListObjectsV2 { public static void main(String[] args) { String accessPointArn = "*** access point ARN ***"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); System.out.println("Listing objects"); // maxKeys is set to 2 to demonstrate the use of // ListObjectsV2Result.getNextContinuationToken() ListObjectsV2Request req = new ListObjectsV2Request().withBucketName(accessPointArn).withMaxKeys(2); ListObjectsV2Result result; do { result = s3Client.listObjectsV2(req); for (S3ObjectSummary objectSummary : result.getObjectSummaries()) { System.out.printf(" - %s (size: %d)\n", objectSummary.getKey(), objectSummary.getSize()); } // If there are more than maxKeys keys in the bucket, get a continuation token // and list the next objects. String token = result.getNextContinuationToken(); System.out.println("Next Continuation Token: " + token); req.setContinuationToken(token); } while (result.isTruncated()); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` # Deleting objects in Amazon S3 on Outposts buckets Objects are the fundamental entities stored in Amazon S3 on Outposts. Every object is contained in a bucket. You must use access points to access any object in an Outpost bucket. When you specify the bucket for object operations, you use the access point Amazon Resource Name (ARN) or the access point alias. For more information about access point aliases, see [Using a bucket-style alias for your S3 on Outposts bucket access point](s3-outposts-access-points-alias.md). The following example shows the ARN format for S3 on Outposts access points, which includes the AWS Region code for the Region that the Outpost is homed to, the AWS account ID, the Outpost ID, and the access point name: ``` arn:aws:s3-outposts:region:account-id:outpost/outpost-id/accesspoint/accesspoint-name ``` For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). With Amazon S3 on Outposts, object data is always stored on the Outpost. When AWS installs an Outpost rack, your data stays local to your Outpost to meet data-residency requirements. Your objects never leave your Outpost and are not in an AWS Region. Because the AWS Management Console is hosted in-Region, you can't use the console to upload or manage objects in your Outpost. However, you can use the REST API, AWS Command Line Interface (AWS CLI), and AWS SDKs to upload and manage your objects through your access points. The following examples show you how to delete a single object or multiple objects in an S3 on Outposts bucket by using the AWS Command Line Interface (AWS CLI) and AWS SDK for Java. ## Using the AWS CLI The following examples show you how to delete a single object or multiple objects from an S3 on Outposts bucket. ------ #### [ delete-object ] The following example deletes an object named `sample-object.xml` from an S3 on Outposts bucket (`s3-outposts:DeleteObject`) by using the AWS CLI. To use this command, replace each `user input placeholder` with your own information. For more information about this command, see [delete-object](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-object.html) in the *AWS CLI Reference*. ``` aws s3api delete-object --bucket arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-outposts-access-point --key sample-object.xml ``` ------ #### [ delete-objects ] The following example deletes two objects named `sample-object.xml` and `test1.text` from an S3 on Outposts bucket (`s3-outposts:DeleteObject`) by using the AWS CLI. To use this command, replace each `user input placeholder` with your own information. For more information about this command, see [delete-objects](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-objects.html) in the *AWS CLI Reference*. ``` aws s3api delete-objects --bucket arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-outposts-access-point --delete file://delete.json delete.json { "Objects": [ { "Key": "test1.txt" }, { "Key": "sample-object.xml" } ], "Quiet": false } ``` ------ ## Using the AWS SDK for Java The following examples show you how to delete a single object or multiple objects from an S3 on Outposts bucket. ------ #### [ DeleteObject ] The following S3 on Outposts example deletes an object in a bucket by using the SDK for Java. To use this example, specify the access point ARN for the Outpost and the key name for the object that you want to delete. For more information, see [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) in the *Amazon Simple Storage Service API Reference*. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.DeleteObjectRequest; public class DeleteObject { public static void main(String[] args) { String accessPointArn = "*** access point ARN ***"; String keyName = "*** key name ****"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); s3Client.deleteObject(new DeleteObjectRequest(accessPointArn, keyName)); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` ------ #### [ DeleteObjects ] The following S3 on Outposts example uploads and then deletes objects in a bucket by using the SDK for Java. To use this example, specify the access point ARN for the Outpost. For more information, see [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) in the *Amazon Simple Storage Service API Reference*. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.DeleteObjectsRequest; import com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion; import com.amazonaws.services.s3.model.DeleteObjectsResult; import java.util.ArrayList; public class DeleteObjects { public static void main(String[] args) { String accessPointArn = "arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-outposts-access-point"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); // Upload three sample objects. ArrayList keys = new ArrayList(); for (int i = 0; i < 3; i++) { String keyName = "delete object example " + i; s3Client.putObject(accessPointArn, keyName, "Object number " + i + " to be deleted."); keys.add(new KeyVersion(keyName)); } System.out.println(keys.size() + " objects successfully created."); // Delete the sample objects. DeleteObjectsRequest multiObjectDeleteRequest = new DeleteObjectsRequest(accessPointArn) .withKeys(keys) .withQuiet(false); // Verify that the objects were deleted successfully. DeleteObjectsResult delObjRes = s3Client.deleteObjects(multiObjectDeleteRequest); int successfulDeletes = delObjRes.getDeletedObjects().size(); System.out.println(successfulDeletes + " objects successfully deleted."); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` ------ # Using HeadBucket to determine if an S3 on Outposts bucket exists and you have access permissions Objects are the fundamental entities stored in Amazon S3 on Outposts. Every object is contained in a bucket. You must use access points to access any object in an Outpost bucket. When you specify the bucket for object operations, you use the access point Amazon Resource Name (ARN) or the access point alias. For more information about access point aliases, see [Using a bucket-style alias for your S3 on Outposts bucket access point](s3-outposts-access-points-alias.md). The following example shows the ARN format for S3 on Outposts access points, which includes the AWS Region code for the Region that the Outpost is homed to, the AWS account ID, the Outpost ID, and the access point name: ``` arn:aws:s3-outposts:region:account-id:outpost/outpost-id/accesspoint/accesspoint-name ``` For more information about S3 on Outposts ARNs, see [Resource ARNs for S3 on Outposts](S3OutpostsIAM.md#S3OutpostsARN). **Note** With Amazon S3 on Outposts, object data is always stored on the Outpost. When AWS installs an Outpost rack, your data stays local to your Outpost to meet data-residency requirements. Your objects never leave your Outpost and are not in an AWS Region. Because the AWS Management Console is hosted in-Region, you can't use the console to upload or manage objects in your Outpost. However, you can use the REST API, AWS Command Line Interface (AWS CLI), and AWS SDKs to upload and manage your objects through your access points. The following AWS Command Line Interface (AWS CLI) and AWS SDK for Java examples show you how to use the HeadBucket API operation to determine if an Amazon S3 on Outposts bucket exists and whether you have permission to access it. For more information, see [HeadBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) in the *Amazon Simple Storage Service API Reference*. ## Using the AWS CLI The following S3 on Outposts AWS CLI example uses the `head-bucket` command to determine if a bucket exists and you have permissions to access it. To use this command, replace each `user input placeholder` with your own information. For more information about this command, see [head-bucket](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/head-bucket.html) in the *AWS CLI Reference*. ``` aws s3api head-bucket --bucket arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/accesspoint/example-outposts-access-point ``` ## Using the AWS SDK for Java The following S3 on Outposts example shows how to determine if a bucket exists and if you have permission to access it. To use this example, specify the access point ARN for the Outpost. For more information, see [HeadBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) in the *Amazon Simple Storage Service API Reference*. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.HeadBucketRequest; public class HeadBucket { public static void main(String[] args) { String accessPointArn = "*** access point ARN ***"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); s3Client.headBucket(new HeadBucketRequest(accessPointArn)); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` # Performing and managing a multipart upload with the SDK for Java With Amazon S3 on Outposts, you can create S3 buckets on your AWS Outposts resources and store and retrieve objects on-premises for applications that require local data access, local data processing, and data residency. You can use S3 on Outposts through the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS SDKs, or REST API. For more information, see [What is Amazon S3 on Outposts?](S3onOutposts.md) The following examples show how you can use S3 on Outposts with the AWS SDK for Java to perform and manage a multipart upload. **Topics** + [ ## Perform a multipart upload of an object in an S3 on Outposts bucket ](#S3OutpostsInitiateMultipartUploadJava) + [ ## Copy a large object in an S3 on Outposts bucket by using multipart upload ](#S3OutpostsCopyPartJava) + [ ## List parts of an object in an S3 on Outposts bucket ](#S3OutpostsListPartsJava) + [ ## Retrieve a list of in-progress multipart uploads in an S3 on Outposts bucket ](#S3OutpostsListMultipartUploadsJava) ## Perform a multipart upload of an object in an S3 on Outposts bucket The following S3 on Outposts example initiates, uploads, and finishes a multipart upload of an object to a bucket by using the SDK for Java. To use this example, replace each `user input placeholder` with your own information. For more information, see [Uploading an object using multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-upload-object.html) in the *Amazon Simple Storage Service User Guide*. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.*; import java.util.ArrayList; import java.util.List; public class MultipartUploadCopy { public static void main(String[] args) { String accessPointArn = "*** Source access point ARN ***"; String sourceObjectKey = "*** Source object key ***"; String destObjectKey = "*** Target object key ***"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); // Initiate the multipart upload. InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(accessPointArn, destObjectKey); InitiateMultipartUploadResult initResult = s3Client.initiateMultipartUpload(initRequest); // Get the object size to track the end of the copy operation. GetObjectMetadataRequest metadataRequest = new GetObjectMetadataRequest(accessPointArn, sourceObjectKey); ObjectMetadata metadataResult = s3Client.getObjectMetadata(metadataRequest); long objectSize = metadataResult.getContentLength(); // Copy the object using 5 MB parts. long partSize = 5 * 1024 * 1024; long bytePosition = 0; int partNum = 1; List copyResponses = new ArrayList(); while (bytePosition < objectSize) { // The last part might be smaller than partSize, so check to make sure // that lastByte isn't beyond the end of the object. long lastByte = Math.min(bytePosition + partSize - 1, objectSize - 1); // Copy this part. CopyPartRequest copyRequest = new CopyPartRequest() .withSourceBucketName(accessPointArn) .withSourceKey(sourceObjectKey) .withDestinationBucketName(accessPointArn) .withDestinationKey(destObjectKey) .withUploadId(initResult.getUploadId()) .withFirstByte(bytePosition) .withLastByte(lastByte) .withPartNumber(partNum++); copyResponses.add(s3Client.copyPart(copyRequest)); bytePosition += partSize; } // Complete the upload request to concatenate all uploaded parts and make the copied object available. CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest( accessPointArn, destObjectKey, initResult.getUploadId(), getETags(copyResponses)); s3Client.completeMultipartUpload(completeRequest); System.out.println("Multipart copy complete."); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } // This is a helper function to construct a list of ETags. private static List getETags(List responses) { List etags = new ArrayList(); for (CopyPartResult response : responses) { etags.add(new PartETag(response.getPartNumber(), response.getETag())); } return etags; } ``` ## Copy a large object in an S3 on Outposts bucket by using multipart upload The following S3 on Outposts example uses the SDK for Java to copy an object in a bucket. To use this example, replace each `user input placeholder` with your own information. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.*; import java.util.ArrayList; import java.util.List; public class MultipartUploadCopy { public static void main(String[] args) { String accessPointArn = "*** Source access point ARN ***"; String sourceObjectKey = "*** Source object key ***"; String destObjectKey = "*** Target object key ***"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); // Initiate the multipart upload. InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(accessPointArn, destObjectKey); InitiateMultipartUploadResult initResult = s3Client.initiateMultipartUpload(initRequest); // Get the object size to track the end of the copy operation. GetObjectMetadataRequest metadataRequest = new GetObjectMetadataRequest(accessPointArn, sourceObjectKey); ObjectMetadata metadataResult = s3Client.getObjectMetadata(metadataRequest); long objectSize = metadataResult.getContentLength(); // Copy the object using 5 MB parts. long partSize = 5 * 1024 * 1024; long bytePosition = 0; int partNum = 1; List copyResponses = new ArrayList(); while (bytePosition < objectSize) { // The last part might be smaller than partSize, so check to make sure // that lastByte isn't beyond the end of the object. long lastByte = Math.min(bytePosition + partSize - 1, objectSize - 1); // Copy this part. CopyPartRequest copyRequest = new CopyPartRequest() .withSourceBucketName(accessPointArn) .withSourceKey(sourceObjectKey) .withDestinationBucketName(accessPointArn) .withDestinationKey(destObjectKey) .withUploadId(initResult.getUploadId()) .withFirstByte(bytePosition) .withLastByte(lastByte) .withPartNumber(partNum++); copyResponses.add(s3Client.copyPart(copyRequest)); bytePosition += partSize; } // Complete the upload request to concatenate all uploaded parts and make the copied object available. CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest( accessPointArn, destObjectKey, initResult.getUploadId(), getETags(copyResponses)); s3Client.completeMultipartUpload(completeRequest); System.out.println("Multipart copy complete."); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } // This is a helper function to construct a list of ETags. private static List getETags(List responses) { List etags = new ArrayList(); for (CopyPartResult response : responses) { etags.add(new PartETag(response.getPartNumber(), response.getETag())); } return etags; } } ``` ## List parts of an object in an S3 on Outposts bucket The following S3 on Outposts example lists the parts of an object in a bucket by using the SDK for Java. To use this example, replace each `user input placeholder` with your own information. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.*; import java.util.List; public class ListParts { public static void main(String[] args) { String accessPointArn = "*** access point ARN ***"; String keyName = "*** Key name ***"; String uploadId = "*** Upload ID ***"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); ListPartsRequest listPartsRequest = new ListPartsRequest(accessPointArn, keyName, uploadId); PartListing partListing = s3Client.listParts(listPartsRequest); List partSummaries = partListing.getParts(); System.out.println(partSummaries.size() + " multipart upload parts"); for (PartSummary p : partSummaries) { System.out.println("Upload part: Part number = \"" + p.getPartNumber() + "\", ETag = " + p.getETag()); } } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` ## Retrieve a list of in-progress multipart uploads in an S3 on Outposts bucket The following S3 on Outposts example shows how to retrieve a list of the in-progress multipart uploads from an Outposts bucket by using the SDK for Java. To use this example, replace each `user input placeholder` with your own information. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.SdkClientException; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.ListMultipartUploadsRequest; import com.amazonaws.services.s3.model.MultipartUpload; import com.amazonaws.services.s3.model.MultipartUploadListing; import java.util.List; public class ListMultipartUploads { public static void main(String[] args) { String accessPointArn = "*** access point ARN ***"; try { // This code expects that you have AWS credentials set up per: // https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .enableUseArnRegion() .build(); // Retrieve a list of all in-progress multipart uploads. ListMultipartUploadsRequest allMultipartUploadsRequest = new ListMultipartUploadsRequest(accessPointArn); MultipartUploadListing multipartUploadListing = s3Client.listMultipartUploads(allMultipartUploadsRequest); List uploads = multipartUploadListing.getMultipartUploads(); // Display information about all in-progress multipart uploads. System.out.println(uploads.size() + " multipart upload(s) in progress."); for (MultipartUpload u : uploads) { System.out.println("Upload in progress: Key = \"" + u.getKey() + "\", id = " + u.getUploadId()); } } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` # Using presigned URLs for S3 on Outposts To grant time-limited access to objects that are stored locally on an Outpost without updating your bucket policy, you can use a presigned URL. With presigned URLs, you as the bucket owner can share objects with individuals in your virtual private cloud (VPC) or grant them the ability to upload or delete objects. When you create a presigned URL by using the AWS SDKs or the AWS Command Line Interface (AWS CLI), you associate the URL with a specific action. You also grant time-limited access to the presigned URL by choosing a custom expiration time that can be as low as 1 second and as high as 7 days. When you share the presigned URL, the individual in the VPC can perform the action embedded in the URL as if they were the original signing user. When the URL reaches its expiration time, the URL expires and no longer works. ## Limiting presigned URL capabilities The capabilities of a presigned URL are limited by the permissions of the user who created it. In essence, presigned URLs are bearer tokens that grant access to those who possess them. As such, we recommend that you protect them appropriately. **AWS Signature Version 4 (SigV4)** To enforce specific behavior when presigned URL requests are authenticated by using AWS Signature Version 4 (SigV4), you can use condition keys in bucket policies and access point policies. For example, you can create a bucket policy that uses the `s3-outposts:signatureAge` condition to deny any Amazon S3 on Outposts presigned URL request on objects in the `example-outpost-bucket` bucket if the signature is more than 10 minutes old. To use this example, replace the *`user input placeholders`* with your own information. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "Deny a presigned URL request if the signature is more than 10 minutes old", "Effect": "Deny", "Principal": {"AWS":"444455556666"}, "Action": "s3-outposts:*", "Resource": "arn:aws:s3-outposts:us-east-1:111122223333:outpost/op-01ac5d28a6a232904/bucket/example-outpost-bucket/object/*", "Condition": { "NumericGreaterThan": {"s3-outposts:signatureAge": 600000}, "StringEquals": {"s3-outposts:authType": "REST-QUERY-STRING"} } } ] } ``` ------ For a list of condition keys and additional example policies that you can use to enforce specific behavior when presigned URL requests are authenticated by using Signature Version 4, see [AWS Signature Version 4 (SigV4) authentication-specific policy keys](s3-outposts-bucket-policy-s3-sigv4-conditions.md). **Network path restriction** If you want to restrict the use of presigned URLs and all S3 on Outposts access to particular network paths, you can write policies that require a particular network path. To set the restriction on the IAM principal that makes the call, you can use identity-based AWS Identity and Access Management (IAM) policies (for example, user, group, or role policies). To set the restriction on the S3 on Outposts resource, you can use resource-based policies (for example, bucket and access point policies). A network-path restriction on the IAM principal requires the user of those credentials to make requests from the specified network. A restriction on the bucket or access point requires that all requests to that resource originate from the specified network. These restrictions also apply outside of the presigned URL scenario. The IAM global condition that you use depends on the type of endpoint. If you are using the public endpoint for S3 on Outposts, use `aws:SourceIp`. If you are using a VPC endpoint for S3 on Outposts, use `aws:SourceVpc` or `aws:SourceVpce`. The following IAM policy statement requires the principal to access AWS only from the specified network range. With this policy statement, all access must originate from that range. This includes the case of someone who's using a presigned URL for S3 on Outposts. To use this example, replace the *`user input placeholders`* with your own information. ``` { "Sid": "NetworkRestrictionForIAMPrincipal", "Effect": "Deny", "Action": "*", "Resource": "*", "Condition": { "NotIpAddressIfExists": {"aws:SourceIp": "IP-address-range"}, "BoolIfExists": {"aws:ViaAWSService": "false"} } } ``` For an example bucket policy that uses the `aws:SourceIP` AWS global condition key to restrict access to an S3 on Outposts bucket to a specific network range, see [Setting up IAM with S3 on Outposts](S3OutpostsIAM.md). ## Who can create a presigned URL Anyone with valid security credentials can create a presigned URL. But for a user in the VPC to successfully access an object, the presigned URL must be created by someone who has permission to perform the operation that the presigned URL is based upon. You can use the following credentials to create a presigned URL: + **IAM instance profile** – Valid up to 6 hours. + **AWS Security Token Service** – Valid up to 36 hours when signed with permanent credentials, such as the credentials of the AWS account root user or an IAM user. + **IAM user** – Valid up to 7 days when you're using AWS Signature Version 4. To create a presigned URL that's valid for up to 7 days, first delegate IAM user credentials (the access key and secret key) to the SDK that you're using. Then, generate a presigned URL by using AWS Signature Version 4. **Note** If you created a presigned URL by using a temporary token, the URL expires when the token expires, even if you created the URL with a later expiration time. Because presigned URLs grant access to your S3 on Outposts buckets to whoever has the URL, we recommend that you protect them appropriately. For more information about protecting presigned URLs, see [Limiting presigned URL capabilities](#S3OutpostsPresignedUrlUploadObjectLimitCapabilities). ## When does S3 on Outposts check the expiration date and time of a presigned URL? At the time of the HTTP request, S3 on Outposts checks the expiration date and time of a signed URL. For example, if a client begins to download a large file immediately before the expiration time, the download continues even if the expiration time passes during the download. However, if the connection drops and the client tries to restart the download after the expiration time passes, the download fails. For more information about using a presigned URL to share or upload objects, see the following topics. **Topics** + [ ## Limiting presigned URL capabilities ](#S3OutpostsPresignedUrlUploadObjectLimitCapabilities) + [ ## Who can create a presigned URL ](#S3Outpostswho-presigned-url) + [ ## When does S3 on Outposts check the expiration date and time of a presigned URL? ](#S3Outpostspresigned-url-when-checked) + [ # Sharing objects by using presigned URLs ](S3OutpostsShareObjectPresignedURL.md) + [ # Generating a presigned URL to upload an object to an S3 on Outposts bucket ](S3OutpostsPresignedUrlUploadObject.md) # Sharing objects by using presigned URLs To grant time-limited access to objects that are stored locally on an Outpost without updating your bucket policy, you can use a presigned URL. With presigned URLs, you as the bucket owner can share objects with individuals in your virtual private cloud (VPC) or grant them the ability to upload or delete objects. When you create a presigned URL by using the AWS SDKs or the AWS Command Line Interface (AWS CLI), you associate the URL with a specific action. You also grant time-limited access to the presigned URL by choosing a custom expiration time that can be as low as 1 second and as high as 7 days. When you share the presigned URL, the individual in the VPC can perform the action embedded in the URL as if they were the original signing user. When the URL reaches its expiration time, the URL expires and no longer works. When you create a presigned URL, you must provide your security credentials, and then specify the following: + An access point Amazon Resource Name (ARN) for the Amazon S3 on Outposts bucket + An object key + An HTTP method (`GET` for downloading objects) + An expiration date and time A presigned URL is valid only for the specified duration. That is, you must start the action that's allowed by the URL before the expiration date and time. You can use a presigned URL multiple times, up to the expiration date and time. If you created a presigned URL by using a temporary token, then the URL expires when the token expires, even if you created the URL with a later expiration time. Users in the virtual private cloud (VPC) who have access to the presigned URL can access the object. For example, if you have a video in your bucket and both the bucket and the object are private, you can share the video with others by generating a presigned URL. Because presigned URLs grant access to your S3 on Outposts buckets to whoever has the URL, we recommend that you protect these URLs appropriately. For more details about protecting presigned URLs, see [Limiting presigned URL capabilities](S3OutpostsPresignedURL.md#S3OutpostsPresignedUrlUploadObjectLimitCapabilities). Anyone with valid security credentials can create a presigned URL. However, the presigned URL must be created by someone who has permission to perform the operation that the presigned URL is based upon. For more information, see [Who can create a presigned URL](S3OutpostsPresignedURL.md#S3Outpostswho-presigned-url). You can generate a presigned URL to share an object in an S3 on Outposts bucket by using the AWS SDKs and the AWS CLI. For more information, see the following examples. ## Using the AWS SDKs You can use the AWS SDKs to generate a presigned URL that you can give to others so that they can retrieve an object. **Note** When you use the AWS SDKs to generate a presigned URL, the maximum expiration time for a presigned URL is 7 days from the time of creation. ------ #### [ Java ] **Example** The following example generates a presigned URL that you can give to others so that they can retrieve an object from an S3 on Outposts bucket. For more information, see [Using presigned URLs for S3 on Outposts](S3OutpostsPresignedURL.md). To use this example, replace the *`user input placeholders`* with your own information. ``` import com.amazonaws.AmazonServiceException; import com.amazonaws.HttpMethod; import com.amazonaws.SdkClientException; import com.amazonaws.auth.profile.ProfileCredentialsProvider; import com.amazonaws.regions.Regions; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest; import java.io.IOException; import java.net.URL; import java.time.Instant; public class GeneratePresignedURL { public static void main(String[] args) throws IOException { Regions clientRegion = Regions.DEFAULT_REGION; String accessPointArn = "*** access point ARN ***"; String objectKey = "*** object key ***"; try { AmazonS3 s3Client = AmazonS3ClientBuilder.standard() .withRegion(clientRegion) .withCredentials(new ProfileCredentialsProvider()) .build(); // Set the presigned URL to expire after one hour. java.util.Date expiration = new java.util.Date(); long expTimeMillis = Instant.now().toEpochMilli(); expTimeMillis += 1000 * 60 * 60; expiration.setTime(expTimeMillis); // Generate the presigned URL. System.out.println("Generating pre-signed URL."); GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(accessPointArn, objectKey) .withMethod(HttpMethod.GET) .withExpiration(expiration); URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest); System.out.println("Pre-Signed URL: " + url.toString()); } catch (AmazonServiceException e) { // The call was transmitted successfully, but Amazon S3 couldn't process // it, so it returned an error response. e.printStackTrace(); } catch (SdkClientException e) { // Amazon S3 couldn't be contacted for a response, or the client // couldn't parse the response from Amazon S3. e.printStackTrace(); } } } ``` ------ #### [ .NET ] **Example** The following example generates a presigned URL that you can give to others so that they can retrieve an object from an S3 on Outposts bucket. For more information, see [Using presigned URLs for S3 on Outposts](S3OutpostsPresignedURL.md). To use this example, replace the *`user input placeholders`* with your own information. ``` using Amazon; using Amazon.S3; using Amazon.S3.Model; using System; namespace Amazon.DocSamples.S3 { class GenPresignedURLTest { private const string accessPointArn = "*** access point ARN ***"; private const string objectKey = "*** object key ***"; // Specify how long the presigned URL lasts, in hours. private const double timeoutDuration = 12; // Specify your bucket Region (an example Region is shown). private static readonly RegionEndpoint bucketRegion = RegionEndpoint.USWest2; private static IAmazonS3 s3Client; public static void Main() { s3Client = new AmazonS3Client(bucketRegion); string urlString = GeneratePreSignedURL(timeoutDuration); } static string GeneratePreSignedURL(double duration) { string urlString = ""; try { GetPreSignedUrlRequest request1 = new GetPreSignedUrlRequest { BucketName = accessPointArn, Key = objectKey, Expires = DateTime.UtcNow.AddHours(duration) }; urlString = s3Client.GetPreSignedURL(request1); } catch (AmazonS3Exception e) { Console.WriteLine("Error encountered on server. Message:'{0}' when writing an object", e.Message); } catch (Exception e) { Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message); } return urlString; } } } ``` ------ #### [ Python ] The following example generates a presigned URL to share an object by using the SDK for Python (Boto3). For example, use a Boto3 client and the `generate_presigned_url` function to generate a presigned URL that allows you to `GET` an object. ``` import boto3 url = boto3.client('s3').generate_presigned_url( ClientMethod='get_object', Params={'Bucket': 'ACCESS_POINT_ARN', 'Key': 'OBJECT_KEY'}, ExpiresIn=3600) ``` For more information about using the SDK for Python (Boto3) to generate a presigned URL, see [Python](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_url) in the *AWS SDK for Python (Boto) API Reference*. ------ ## Using the AWS CLI The following example AWS CLI command generates a presigned URL for an S3 on Outposts bucket. To use this example, replace the *`user input placeholders`* with your own information. **Note** When you use the AWS CLI to generate a presigned URL, the maximum expiration time for a presigned URL is 7 days from the time of creation. ``` aws s3 presign s3://arn:aws:s3-outposts:us-east-1:111122223333:outpost/op-01ac5d28a6a232904/accesspoint/example-outpost-access-point/mydoc.txt --expires-in 604800 ``` For more information, see [presign](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html) in the *AWS CLI Command Reference*. # Generating a presigned URL to upload an object to an S3 on Outposts bucket To grant time-limited access to objects that are stored locally on an Outpost without updating your bucket policy, you can use a presigned URL. With presigned URLs, you as the bucket owner can share objects with individuals in your virtual private cloud (VPC) or grant them the ability to upload or delete objects. When you create a presigned URL by using the AWS SDKs or the AWS Command Line Interface (AWS CLI), you associate the URL with a specific action. You also grant time-limited access to the presigned URL by choosing a custom expiration time that can be as low as 1 second and as high as 7 days. When you share the presigned URL, the individual in the VPC can perform the action embedded in the URL as if they were the original signing user. When the URL reaches its expiration time, the URL expires and no longer works. When you create a presigned URL, you must provide your security credentials, and then specify the following: + An access point Amazon Resource Name (ARN) for the Amazon S3 on Outposts bucket + An object key + An HTTP method (`PUT` for uploading objects) + An expiration date and time A presigned URL is valid only for the specified duration. That is, you must start the action that's allowed by the URL before the expiration date and time. You can use a presigned URL multiple times, up to the expiration date and time. If you created a presigned URL by using a temporary token, then the URL expires when the token expires, even if you created the URL with a later expiration time. If the action allowed by a presigned URL consists of multiple steps, such as a multipart upload, you must start all steps before the expiration time. If S3 on Outposts tries to start a step with an expired URL, you receive an error. Users in the virtual private cloud (VPC) who have access to the presigned URL can upload objects. For example, a user in the VPC who has access to the presigned URL can upload an object to your bucket. Because presigned URLs grant access to your S3 on Outposts bucket to any user in the VPC who has access to the presigned URL, we recommend that you protect these URLs appropriately. For more details about protecting presigned URLs, see [Limiting presigned URL capabilities](S3OutpostsPresignedURL.md#S3OutpostsPresignedUrlUploadObjectLimitCapabilities). Anyone with valid security credentials can create a presigned URL. However, the presigned URL must be created by someone who has permission to perform the operation that the presigned URL is based upon. For more information, see [Who can create a presigned URL](S3OutpostsPresignedURL.md#S3Outpostswho-presigned-url). ## Using the AWS SDKs to generate a presigned URL for an S3 on Outposts object operation ------ #### [ Java ] **SDK for Java 2.x** This example shows how to generate a presigned URL that you can use to upload an object to an S3 on Outposts bucket for a limited time. For more information, see [Using presigned URLs for S3 on Outposts](S3OutpostsPresignedURL.md). ``` public static void signBucket(S3Presigner presigner, String outpostAccessPointArn, String keyName) { try { PutObjectRequest objectRequest = PutObjectRequest.builder() .bucket(accessPointArn) .key(keyName) .contentType("text/plain") .build(); PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder() .signatureDuration(Duration.ofMinutes(10)) .putObjectRequest(objectRequest) .build(); PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest); String myURL = presignedRequest.url().toString(); System.out.println("Presigned URL to upload a file to: " +myURL); System.out.println("Which HTTP method must be used when uploading a file: " + presignedRequest.httpRequest().method()); // Upload content to the S3 on Outposts bucket by using this URL. URL url = presignedRequest.url(); // Create the connection and use it to upload the new object by using the presigned URL. HttpURLConnection connection = (HttpURLConnection) url.openConnection(); connection.setDoOutput(true); connection.setRequestProperty("Content-Type","text/plain"); connection.setRequestMethod("PUT"); OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream()); out.write("This text was uploaded as an object by using a presigned URL."); out.close(); connection.getResponseCode(); System.out.println("HTTP response code is " + connection.getResponseCode()); } catch (S3Exception e) { e.getStackTrace(); } catch (IOException e) { e.getStackTrace(); } } ``` ------ #### [ Python ] **SDK for Python (Boto3)** This example shows how to generate a presigned URL that can perform an S3 on Outposts action for a limited time. For more information, see [Using presigned URLs for S3 on Outposts](S3OutpostsPresignedURL.md). To make a request with the URL, use the `Requests` package. ``` import argparse import logging import boto3 from botocore.exceptions import ClientError import requests logger = logging.getLogger(__name__) def generate_presigned_url(s3_client, client_method, method_parameters, expires_in): """ Generate a presigned S3 on Outposts URL that can be used to perform an action. :param s3_client: A Boto3 Amazon S3 client. :param client_method: The name of the client method that the URL performs. :param method_parameters: The parameters of the specified client method. :param expires_in: The number of seconds that the presigned URL is valid for. :return: The presigned URL. """ try: url = s3_client.generate_presigned_url( ClientMethod=client_method, Params=method_parameters, ExpiresIn=expires_in ) logger.info("Got presigned URL: %s", url) except ClientError: logger.exception( "Couldn't get a presigned URL for client method '%s'.", client_method) raise return url def usage_demo(): logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s') print('-'*88) print("Welcome to the Amazon S3 on Outposts presigned URL demo.") print('-'*88) parser = argparse.ArgumentParser() parser.add_argument('accessPointArn', help="The name of the S3 on Outposts access point ARN.") parser.add_argument( 'key', help="For a GET operation, the key of the object in S3 on Outposts. For a " "PUT operation, the name of a file to upload.") parser.add_argument( 'action', choices=('get', 'put'), help="The action to perform.") args = parser.parse_args() s3_client = boto3.client('s3') client_action = 'get_object' if args.action == 'get' else 'put_object' url = generate_presigned_url( s3_client, client_action, {'Bucket': args.accessPointArn, 'Key': args.key}, 1000) print("Using the Requests package to send a request to the URL.") response = None if args.action == 'get': response = requests.get(url) elif args.action == 'put': print("Putting data to the URL.") try: with open(args.key, 'r') as object_file: object_text = object_file.read() response = requests.put(url, data=object_text) except FileNotFoundError: print(f"Couldn't find {args.key}. For a PUT operation, the key must be the " f"name of a file that exists on your computer.") if response is not None: print("Got response:") print(f"Status: {response.status_code}") print(response.text) print('-'*88) if __name__ == '__main__': usage_demo() ``` ------ # Amazon S3 on Outposts with local Amazon EMR on Outposts Amazon EMR is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark, on AWS to process and analyze vast amounts of data. By using these frameworks and related open-source projects, you can process data for analytics purposes and business intelligence workloads. Amazon EMR also helps you transform and move large amounts of data into and out of other AWS data stores and databases, and supports Amazon S3 on Outposts. For more information about Amazon EMR, see [Amazon EMR on Outposts](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-outposts.html) in the *Amazon EMR Management Guide*. For Amazon S3 on Outposts, Amazon EMR started to support the Apache Hadoop S3A connector in version 7.0.0. Earlier versions of Amazon EMR don't support local S3 on Outposts, and the EMR File System (EMRFS) is not supported. **Supported applications** Amazon EMR with Amazon S3 on Outposts supports the following applications: + Hadoop + Spark + Hue + Hive + Sqoop + Pig + Hudi + Flink For more information, see the [Amazon EMR Release Guide](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html). ## Create and configure an Amazon S3 on Outposts bucket Amazon EMR uses the AWS SDK for Java with Amazon S3 on Outposts to store input data and output data. Your Amazon EMR log files are stored in a Regional Amazon S3 location that you select and aren't stored locally on the Outpost. For more information, see [Amazon EMR logs](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-view-web-log-files.html) in the *Amazon EMR Management Guide*. To conform with Amazon S3 and DNS requirements, S3 on Outposts buckets have naming restrictions and limitations. For more information, see [Creating an S3 on Outposts bucket](S3OutpostsCreateBucket.md). With Amazon EMR version 7.0.0 and later, you can use Amazon EMR with S3 on Outposts and the S3A file system. **Prerequisites** **S3 on Outposts permissions** – When you create your Amazon EMR instance profile, your role must contain the AWS Identity and Access Management (IAM) namespace for S3 on Outposts. S3 on Outposts has its own namespace, `s3-outposts*`. For an example policy that uses this namespace, see [Setting up IAM with S3 on Outposts](S3OutpostsIAM.md). **S3A connector** – To configure your EMR cluster to access data from an Amazon S3 on Outposts bucket, you must use the Apache Hadoop S3A connector. To use the connector, ensure that all of your S3 URIs use the `s3a` scheme. If they don't, you can configure the file system implementation that you use for your EMR cluster so that your S3 URIs work with the S3A connector. To configure the file system implementation to work with the S3A connector, you use the `fs.file_scheme.impl` and `fs.AbstractFileSystem.file_scheme.impl` configuration properties for your EMR cluster, where `file_scheme` corresponds to the type of S3 URIs that you have. To use the following example, replace the *`user input placeholders`* with your own information. For example, to change the file system implementation for S3 URIs that use the `s3` scheme, specify the following cluster configuration properties: ``` 1. [ 2. { 3. "Classification": "core-site", 4. "Properties": { 5. "fs.s3.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem", 6. "fs.AbstractFileSystem.s3.impl": "org.apache.hadoop.fs.s3a.S3A" 7. } 8. } 9. ] ``` To use S3A, set the `fs.file_scheme.impl` configuration property to `org.apache.hadoop.fs.s3a.S3AFileSystem`, and set the `fs.AbstractFileSystem.file_scheme.impl` property to `org.apache.hadoop.fs.s3a.S3A`. For example, if you are accessing the path `s3a://bucket/...`, set the `fs.s3a.impl` property to `org.apache.hadoop.fs.s3a.S3AFileSystem`, and set the `fs.AbstractFileSystem.s3a.impl` property to `org.apache.hadoop.fs.s3a.S3A`. ## Getting started using Amazon EMR with Amazon S3 on Outposts The following topics explain how to get started using Amazon EMR with Amazon S3 on Outposts. **Topics** + [ ### Create a permissions policy ](#create-permission-policy) + [ ### Create and configure your cluster ](#configure-cluster) + [ ### Configurations overview ](#configurations-overview) + [ ### Considerations ](#considerations) ### Create a permissions policy Before you can create an EMR cluster that uses Amazon S3 on Outposts, you must create an IAM policy to attach to the Amazon EC2 instance profile for the cluster. The policy must have permissions to access the S3 on Outposts access point Amazon Resource Name (ARN). For more information about creating IAM policies for S3 on Outposts, see [Setting up IAM with S3 on Outposts](S3OutpostsIAM.md). The following example policy shows how to grant the required permissions. After you create the policy, attach the policy to the instance profile role that you use to create your EMR cluster, as described in the [Create and configure your cluster](#configure-cluster) section. To use this example, replace the *`user input placeholders`* with your own information. ``` 1. { 2. "Version":"2012-10-17", 3. "Statement": [ 4. { 5. "Effect": "Allow", 6. "Resource": "arn:aws:s3-outposts:us-west-2:111122223333:outpost/op-01ac5d28a6a232904/accesspoint/access-point-name, 7. "Action": [ 8. "s3-outposts:*" 9. ] 10. } 11. ] 12. 13. } ``` ### Create and configure your cluster To create a cluster that runs Spark with S3 on Outposts, complete the following steps in the console. **To create a cluster that runs Spark with S3 on Outposts** 1. Open the Amazon EMR console at [https://console.aws.amazon.com/elasticmapreduce/](https://console.aws.amazon.com/elasticmapreduce/). 1. In the left navigation pane, choose **Clusters**. 1. Choose **Create cluster**. 1. For **Amazon EMR release**, choose **emr-7.0.0** or later. 1. For Application bundle, choose **Spark interactive**. Then select any other supported applications that you want to be included in your cluster. 1. To enable Amazon S3 on Outposts, enter your configuration settings. **Sample configuration settings** To use the following sample configuration settings, replace the `user input placeholders` with your own information. ``` 1. [ 2. { 3. "Classification": "core-site", 4. "Properties": { 5. "fs.s3a.bucket.DOC-EXAMPLE-BUCKET.accesspoint.arn": "arn:aws:s3-outposts:us-west-2:111122223333:outpost/op-01ac5d28a6a232904/accesspoint/access-point-name" 6. "fs.s3a.committer.name": "magic", 7. "fs.s3a.select.enabled": "false" 8. } 9. }, 10. { 11. "Classification": "hadoop-env", 12. "Configurations": [ 13. { 14. "Classification": "export", 15. "Properties": { 16. "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64" 17. } 18. } 19. ], 20. "Properties": {} 21. }, 22. { 23. "Classification": "spark-env", 24. "Configurations": [ 25. { 26. "Classification": "export", 27. "Properties": { 28. "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64" 29. } 30. } 31. ], 32. "Properties": {} 33. }, 34. { 35. "Classification": "spark-defaults", 36. "Properties": { 37. "spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64", 38. "spark.sql.sources.fastS3PartitionDiscovery.enabled": "false" 39. } 40. } 41. ] ``` 1. In the **Networking** section, choose a virtual private cloud (VPC) and subnet that are on your AWS Outposts rack. For more information about Amazon EMR on Outposts, see [EMR clusters on AWS Outposts](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-outposts.html) in the *Amazon EMR Management Guide*. 1. In the **EC2 instance profile for Amazon EMR** section, choose the IAM role that has the [permissions policy that you created earlier](#create-permission-policy) attached. 1. Configure your remaining cluster settings, and then choose **Create cluster**. ### Configurations overview The following table describes S3A configurations and the values to specify for their parameters when you set up a cluster that uses S3 on Outposts with Amazon EMR. | Parameter | Default value | Required value for S3 on Outposts | Explanation | | --- | --- | --- | --- | | `fs.s3a.aws.credentials.provider` | If not specified, S3A will look for S3 in Region bucket with the Outposts bucket name. | The access point ARN of the S3 on Outposts bucket | Amazon S3 on Outposts supports virtual private cloud (VPC)-only access points as the only means to access your Outposts buckets. | | `fs.s3a.committer.name` | file | `magic` | Magic committer is the only supported committer for S3 on Outposts. | | `fs.s3a.select.enabled` | `TRUE` | `FALSE` | S3 Select is not supported on Outposts. | | `JAVA_HOME` | `/usr/lib/jvm/java-8` | `/usr/lib/jvm/java-11-amazon-corretto.x86_64` | S3 on Outposts on S3A requires Java version 11. | The following table describes Spark configurations and the values to specify for their parameters when you set up a cluster that uses S3 on Outposts with Amazon EMR. | Parameter | Default value | Required value for S3 on Outposts | Explanation | | --- | --- | --- | --- | | `spark.sql.sources.fastS3PartitionDiscovery.enabled` | `TRUE` | `FALSE` | S3 on Outposts doesn't support fast partition. | | `spark.executorEnv.JAVA_HOME` | `/usr/lib/jvm/java-8` | `/usr/lib/jvm/java-11-amazon-corretto.x86_64` | S3 on Outposts on S3A requires Java version 11. | ### Considerations Consider the following when you integrate Amazon EMR with S3 on Outposts buckets: + Amazon S3 on Outposts is supported with Amazon EMR version 7.0.0 and later. + The S3A connector is required to use S3 on Outposts with Amazon EMR. Only S3A has the features required to interact with S3 on Outposts buckets. For S3A connector setup information, see [Prerequisites](#s3a-outposts-prerequisites). + Amazon S3 on Outposts supports only server-side encryption with Amazon S3 managed keys (SSE-S3) with Amazon EMR. For more information, see [Data encryption in S3 on Outposts](s3-outposts-data-encryption.md). + Amazon S3 on Outposts doesn't support writes with the S3A FileOutputCommitter. Writes with the S3A FileOutputCommitter on S3 on Outposts buckets result in the following error: InvalidStorageClass: The storage class you specified is not valid. + Amazon S3 on Outposts isn't supported with Amazon EMR Serverless or Amazon EMR on EKS. + Amazon EMR logs are stored in a Regional Amazon S3 location that you select, and are not stored locally in the S3 on Outposts bucket. # Authorization and authentication caching S3 on Outposts securely caches authentication and authorization data locally on Outposts racks. The cache removes round trips to the parent AWS Region for every request. This eliminates the variability that is introduced by network round trips. With the authentication and authorization cache in S3 on Outposts, you get consistent latencies that are independent from the latency of the connection between the Outposts and the AWS Region. When you make an S3 on Outposts API request, the authentication and authorization data is securely cached. The cached data is then used to authenticate subsequent S3 object API requests. S3 on Outposts only caches authentication and authorization data when the request is signed using Signature Version 4A (SigV4A). The cache is stored locally on the Outposts within the S3 on Outposts service. It asynchronously refreshes when you make an S3 API request. The cache is encrypted, and no plaintext cryptographic keys are stored on Outposts. The cache is valid for up to 10 minutes when the Outpost is connected to the AWS Region. It is refreshed asynchronously when you make an S3 on Outposts API request, to ensure that the latest policies are used. If the Outpost is disconnected from the AWS Region, the cache will be valid for up to 12 hours. ## Configuring the authorization and authentication cache S3 on Outposts automatically caches authentication and authorization data for requests signed with the SigV4A algorithm. For more information, see [Signing AWS API requests](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-signing.html) in the *AWS Identity and Access Management User Guide*. The SigV4A algorithm is available in the latest versions of the AWS SDKs. You can obtain it through a dependency on the [AWS Common Runtime (CRT) libraries](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html). You need to use the latest version of the AWS SDK and install the latest version of the CRT. For example, you can run `pip install awscrt` to obtain the latest version of the CRT with Boto3. S3 on Outposts does not cache authentication and authorization data for requests signed with the SigV4 algorithm. ## Validating SigV4A signing You can use AWS CloudTrail to validate that requests were signed with SigV4A. For more information on setting up CloudTrail for S3 on Outposts, see [Monitoring S3 on Outposts with AWS CloudTrail logs](S3OutpostsCloudtrail.md). After you have configured CloudTrail, you can verify how a request was signed in the `SignatureVersion` field of the CloudTrail logs. Requests that were signed with SigV4A will have a `SignatureVersion` set to `AWS4-ECDSA-P256-SHA256`. Requests that were signed with SigV4 will have `SignatureVersion` set to `AWS4-HMAC-SHA256`.