There are more AWS SDK examples available in the [AWS Doc SDK Examples](https://github.com/awsdocs/aws-doc-sdk-examples) GitHub repo.
# Scenarios for Amazon S3 using AWS SDKs
The following code examples show you how to implement common scenarios in Amazon S3 with AWS SDKs. These scenarios show you how to accomplish specific tasks by calling multiple functions within Amazon S3 or combined with other AWS services. Each scenario includes a link to the complete source code, where you can find instructions on how to set up and run the code.
Scenarios target an intermediate level of experience to help you understand service actions in context.
**Topics**
+ [Check if a bucket exists](s3_example_s3_Scenario_DoesBucketExist_section.md)
+ [Convert text to speech and back to text](s3_example_cross_Telephone_section.md)
+ [Create a presigned URL](s3_example_s3_Scenario_PresignedUrl_section.md)
+ [Create a serverless application to manage photos](s3_example_cross_PAM_section.md)
+ [Create a web page that lists Amazon S3 objects](s3_example_s3_Scenario_ListObjectsWeb_section.md)
+ [Create an Amazon Textract explorer application](s3_example_cross_TextractExplorer_section.md)
+ [Delete all objects in a bucket](s3_example_s3_Scenario_DeleteAllObjects_section.md)
+ [Delete incomplete multipart uploads](s3_example_s3_Scenario_AbortMultipartUpload_section.md)
+ [Detect PPE in images](s3_example_cross_RekognitionPhotoAnalyzerPPE_section.md)
+ [Detect entities in text extracted from an image](s3_example_cross_TextractComprehendDetectEntities_section.md)
+ [Detect faces in an image](s3_example_cross_DetectFaces_section.md)
+ [Detect objects in images](s3_example_cross_RekognitionPhotoAnalyzer_section.md)
+ [Detect people and objects in a video](s3_example_cross_RekognitionVideoDetection_section.md)
+ [Download S3 'directories'](s3_example_s3_Scenario_DownloadS3Directory_section.md)
+ [Download objects to a local directory](s3_example_s3_DownloadBucketToDirectory_section.md)
+ [Download stream of unknown size](s3_example_s3_Scenario_DownloadStream_section.md)
+ [Get an object from a Multi-Region Access Point](s3_example_s3_GetObject_MRAP_section.md)
+ [Get an object from a bucket if it has been modified](s3_example_s3_GetObject_IfModifiedSince_section.md)
+ [Get started with encryption](s3_example_s3_Encryption_section.md)
+ [Get started with tags](s3_example_s3_Scenario_Tagging_section.md)
+ [Getting started with Amazon Athena](s3_example_athena_GettingStarted_061_section.md)
+ [Getting started with Amazon EMR](s3_example_emr_GettingStarted_037_section.md)
+ [Getting started with Amazon S3](s3_example_s3_GettingStarted_section.md)
+ [Getting started with Amazon SageMaker Feature Store](s3_example_iam_GettingStarted_028_section.md)
+ [Getting started with Amazon Textract](s3_example_s3_GettingStarted_074_section.md)
+ [Getting started with Config](s3_example_config_service_GettingStarted_053_section.md)
+ [Lock Amazon S3 objects](s3_example_s3_Scenario_ObjectLock_section.md)
+ [Make conditional requests](s3_example_s3_Scenario_ConditionalRequests_section.md)
+ [Manage access control lists (ACLs)](s3_example_s3_Scenario_ManageACLs_section.md)
+ [Manage large messages using S3](s3_example_sqs_Scenario_SqsExtendedClient_section.md)
+ [Manage versioned objects in batches with a Lambda function](s3_example_s3_Scenario_BatchObjectVersioning_section.md)
+ [Parse URIs](s3_example_s3_Scenario_URIParsing_section.md)
+ [Perform a multipart copy](s3_example_s3_MultipartCopy_section.md)
+ [Process S3 event notifications](s3_example_s3_Scenario_ProcessS3EventNotification_section.md)
+ [Save EXIF and other image information](s3_example_cross_DetectLabels_section.md)
+ [Send event notifications to EventBridge](s3_example_s3_Scenario_PutBucketNotificationConfiguration_section.md)
+ [Track uploads and downloads](s3_example_s3_Scenario_TrackUploadDownload_section.md)
+ [Transform data with S3 Object Lambda](s3_example_cross_ServerlessS3DataTransformation_section.md)
+ [Unit and integration test with an SDK](s3_example_cross_Testing_section.md)
+ [Upload directory to a bucket](s3_example_s3_UploadDirectoryToBucket_section.md)
+ [Upload or download large files](s3_example_s3_Scenario_UsingLargeFiles_section.md)
+ [Upload stream of unknown size](s3_example_s3_Scenario_UploadStream_section.md)
+ [Use checksums](s3_example_s3_Scenario_UseChecksums_section.md)
+ [Work with Amazon S3 object integrity](s3_example_s3_Scenario_ObjectIntegrity_section.md)
+ [Work with versioned objects](s3_example_s3_Scenario_ObjectVersioningUsage_section.md)
# Check if a bucket exists
The following code example shows how to check if a bucket exists.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
You can use the following `doesBucketExists` method as a replacement for the the SDK for Java V1 [AmazonS3Client\$1doesBucketExistV2(String)](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#doesBucketExistV2-java.lang.String-) method.
```
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.awscore.exception.AwsServiceException;
import software.amazon.awssdk.http.HttpStatusCode;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.utils.Validate;
public class DoesBucketExist {
private static final Logger logger = LoggerFactory.getLogger(DoesBucketExist.class);
public static void main(String[] args) {
DoesBucketExist doesBucketExist = new DoesBucketExist();
final S3Client s3SyncClient = S3Client.builder().build();
final String bucketName = "amzn-s3-demo-bucket"; // Change to the bucket name that you want to check.
boolean exists = doesBucketExist.doesBucketExist(bucketName, s3SyncClient);
logger.info("Bucket exists: {}", exists);
}
/**
* Checks if the specified bucket exists. Amazon S3 buckets are named in a global namespace; use this method to
* determine if a specified bucket name already exists, and therefore can't be used to create a new bucket.
*
* Internally this method uses the S3Client.getBucketAcl(String)
* operation to determine whether the bucket exists.
*
* This method is equivalent to the AWS SDK for Java V1's AmazonS3Client#doesBucketExistV2(String).
*
* @param bucketName The name of the bucket to check.
* @param s3SyncClient An S3Client instance. The method checks for the bucket in the AWS Region
* configured on the instance.
* @return The value true if the specified bucket exists in Amazon S3; the value false if there is no bucket in
* Amazon S3 with that name.
*/
public boolean doesBucketExist(String bucketName, S3Client s3SyncClient) {
try {
Validate.notEmpty(bucketName, "The bucket name must not be null or an empty string.", "");
s3SyncClient.getBucketAcl(r -> r.bucket(bucketName));
return true;
} catch (AwsServiceException ase) {
// A redirect error or an AccessDenied exception means the bucket exists but it's not in this region
// or we don't have permissions to it.
if ((ase.statusCode() == HttpStatusCode.MOVED_PERMANENTLY) || "AccessDenied".equals(ase.awsErrorDetails().errorCode())) {
return true;
}
if (ase.statusCode() == HttpStatusCode.NOT_FOUND) {
return false;
}
throw ase;
}
}
}
```
+ For API details, see [GetBucketAcl](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/GetBucketAcl) in *AWS SDK for Java 2.x API Reference*.
------
# Convert text to speech and back to text using an AWS SDK
The following code example shows how to:
+ Use Amazon Polly to synthesize a plain text (UTF-8) input file to an audio file.
+ Upload the audio file to an Amazon S3 bucket.
+ Use Amazon Transcribe to convert the audio file to text.
+ Display the text.
------
#### [ Rust ]
**SDK for Rust**
Use Amazon Polly to synthesize a plain text (UTF-8) input file to an audio file, upload the audio file to an Amazon S3 bucket, use Amazon Transcribe to convert that audio file to text, and display the text.
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/rustv1/cross_service#code-examples).
**Services used in this example**
+ Amazon Polly
+ Amazon S3
+ Amazon Transcribe
------
# Create a presigned URL for Amazon S3 using an AWS SDK
The following code examples show how to create a presigned URL for Amazon S3 and upload an object.
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/#code-examples).
Generate a presigned URL that can perform an Amazon S3 action for a limited time.
```
using System;
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
public class GenPresignedUrl
{
public static void Main()
{
const string bucketName = "amzn-s3-demo-bucket";
const string objectKey = "sample.txt";
// Specify how long the presigned URL lasts, in hours
const double timeoutDuration = 12;
// Specify the AWS Region of your Amazon S3 bucket. If it is
// different from the Region defined for the default user,
// pass the Region to the constructor for the client. For
// example: new AmazonS3Client(RegionEndpoint.USEast1);
// If using the Region us-east-1, and server-side encryption with AWS KMS, you must specify Signature Version 4.
// Region us-east-1 defaults to Signature Version 2 unless explicitly set to Version 4 as shown below.
// For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#specify-signature-version
// and https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/Amazon/TAWSConfigsS3.html
AWSConfigsS3.UseSignatureVersion4 = true;
IAmazonS3 s3Client = new AmazonS3Client(RegionEndpoint.USEast1);
string urlString = GeneratePresignedURL(s3Client, bucketName, objectKey, timeoutDuration);
Console.WriteLine($"The generated URL is: {urlString}.");
}
///
/// Generate a presigned URL that can be used to access the file named
/// in the objectKey parameter for the amount of time specified in the
/// duration parameter.
///
/// An initialized S3 client object used to call
/// the GetPresignedUrl method.
/// The name of the S3 bucket containing the
/// object for which to create the presigned URL.
/// The name of the object to access with the
/// presigned URL.
/// The length of time for which the presigned
/// URL will be valid.
/// A string representing the generated presigned URL.
public static string GeneratePresignedURL(IAmazonS3 client, string bucketName, string objectKey, double duration)
{
string urlString = string.Empty;
try
{
var request = new GetPreSignedUrlRequest()
{
BucketName = bucketName,
Key = objectKey,
Expires = DateTime.UtcNow.AddHours(duration),
};
urlString = client.GetPreSignedURL(request);
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"Error:'{ex.Message}'");
}
return urlString;
}
}
```
Generate a presigned URL and perform an upload using that URL.
```
using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
///
/// This example shows how to upload an object to an Amazon Simple Storage
/// Service (Amazon S3) bucket using a presigned URL. The code first
/// creates a presigned URL and then uses it to upload an object to an
/// Amazon S3 bucket using that URL.
///
public class UploadUsingPresignedURL
{
private static HttpClient httpClient = new HttpClient();
public static async Task Main()
{
string bucketName = "amzn-s3-demo-bucket";
string keyName = "samplefile.txt";
string filePath = $"source\\{keyName}";
// Specify how long the signed URL will be valid in hours.
double timeoutDuration = 12;
// Specify the AWS Region of your Amazon S3 bucket. If it is
// different from the Region defined for the default user,
// pass the Region to the constructor for the client. For
// example: new AmazonS3Client(RegionEndpoint.USEast1);
// If using the Region us-east-1, and server-side encryption with AWS KMS, you must specify Signature Version 4.
// Region us-east-1 defaults to Signature Version 2 unless explicitly set to Version 4 as shown below.
// For more details, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#specify-signature-version
// and https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/Amazon/TAWSConfigsS3.html
AWSConfigsS3.UseSignatureVersion4 = true;
IAmazonS3 client = new AmazonS3Client(RegionEndpoint.USEast1);
var url = GeneratePreSignedURL(client, bucketName, keyName, timeoutDuration);
var success = await UploadObject(filePath, url);
if (success)
{
Console.WriteLine("Upload succeeded.");
}
else
{
Console.WriteLine("Upload failed.");
}
}
///
/// Uploads an object to an Amazon S3 bucket using the presigned URL passed in
/// the url parameter.
///
/// The path (including file name) to the local
/// file you want to upload.
/// The presigned URL that will be used to upload the
/// file to the Amazon S3 bucket.
/// A Boolean value indicating the success or failure of the
/// operation, based on the HttpWebResponse.
public static async Task UploadObject(string filePath, string url)
{
using var streamContent = new StreamContent(
new FileStream(filePath, FileMode.Open, FileAccess.Read));
var response = await httpClient.PutAsync(url, streamContent);
return response.IsSuccessStatusCode;
}
///
/// Generates a presigned URL which will be used to upload an object to
/// an Amazon S3 bucket.
///
/// The initialized Amazon S3 client object used to call
/// GetPreSignedURL.
/// The name of the Amazon S3 bucket to which the
/// presigned URL will point.
/// The name of the file that will be uploaded.
/// How long (in hours) the presigned URL will
/// be valid.
/// The generated URL.
public static string GeneratePreSignedURL(
IAmazonS3 client,
string bucketName,
string objectKey,
double duration)
{
var request = new GetPreSignedUrlRequest
{
BucketName = bucketName,
Key = objectKey,
Verb = HttpVerb.PUT,
Expires = DateTime.UtcNow.AddHours(duration),
};
string url = client.GetPreSignedURL(request);
return url;
}
}
```
**SDK for .NET (v4)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv4/S3/Scenarios/S3_CreatePresignedPost#code-examples).
Create and use presigned POST URLs for direct browser uploads.
```
///
/// Scenario demonstrating the complete workflow for presigned POST URLs:
/// 1. Create an S3 bucket
/// 2. Create a presigned POST URL
/// 3. Upload a file using the presigned POST URL
/// 4. Clean up resources
///
public class CreatePresignedPostBasics
{
public static ILogger _logger = null!;
public static S3Wrapper _s3Wrapper = null!;
public static UiMethods _uiMethods = null!;
public static IHttpClientFactory _httpClientFactory = null!;
public static bool _isInteractive = true;
public static string? _bucketName;
public static string? _objectKey;
///
/// Set up the services and logging.
///
/// The IHost instance.
public static void SetUpServices(IHost host)
{
var loggerFactory = LoggerFactory.Create(builder =>
{
builder.AddConsole();
});
_logger = new Logger(loggerFactory);
_s3Wrapper = host.Services.GetRequiredService();
_httpClientFactory = host.Services.GetRequiredService();
_uiMethods = new UiMethods();
}
///
/// Perform the actions defined for the Amazon S3 Presigned POST scenario.
///
/// Command line arguments.
/// A Task object.
public static async Task Main(string[] args)
{
_isInteractive = !args.Contains("--non-interactive");
// Set up dependency injection for Amazon S3
using var host = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder(args)
.ConfigureServices((_, services) =>
services.AddAWSService()
.AddTransient()
.AddHttpClient()
)
.Build();
SetUpServices(host);
try
{
// Display overview
_uiMethods.DisplayOverview();
_uiMethods.PressEnter(_isInteractive);
// Step 1: Create bucket
await CreateBucketAsync();
_uiMethods.PressEnter(_isInteractive);
// Step 2: Create presigned URL
_uiMethods.DisplayTitle("Step 2: Create presigned POST URL");
var response = await CreatePresignedPostAsync();
_uiMethods.PressEnter(_isInteractive);
// Step 3: Display URL and fields
_uiMethods.DisplayTitle("Step 3: Presigned POST URL details");
DisplayPresignedPostFields(response);
_uiMethods.PressEnter(_isInteractive);
// Step 4: Upload file
_uiMethods.DisplayTitle("Step 4: Upload test file using presigned POST URL");
await UploadFileAsync(response);
_uiMethods.PressEnter(_isInteractive);
// Step 5: Verify file exists
await VerifyFileExistsAsync();
_uiMethods.PressEnter(_isInteractive);
// Step 6: Cleanup
_uiMethods.DisplayTitle("Step 6: Clean up resources");
await CleanupAsync();
_uiMethods.DisplayTitle("S3 Presigned POST Scenario completed successfully!");
_uiMethods.PressEnter(_isInteractive);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error in scenario");
Console.WriteLine($"Error: {ex.Message}");
// Attempt cleanup if there was an error
if (!string.IsNullOrEmpty(_bucketName))
{
_uiMethods.DisplayTitle("Cleaning up resources after error");
await _s3Wrapper.DeleteBucketAsync(_bucketName);
Console.WriteLine($"Cleaned up bucket: {_bucketName}");
}
}
}
///
/// Create an S3 bucket for the scenario.
///
private static async Task CreateBucketAsync()
{
_uiMethods.DisplayTitle("Step 1: Create an S3 bucket");
// Generate a default bucket name for the scenario
var defaultBucketName = $"presigned-post-demo-{DateTime.Now:yyyyMMddHHmmss}".ToLower();
// Prompt user for bucket name or use default in non-interactive mode
_bucketName = _uiMethods.GetUserInput(
$"Enter S3 bucket name (or press Enter for '{defaultBucketName}'): ",
defaultBucketName,
_isInteractive);
// Basic validation to ensure bucket name is not empty
if (string.IsNullOrWhiteSpace(_bucketName))
{
_bucketName = defaultBucketName;
}
Console.WriteLine($"Creating bucket: {_bucketName}");
await _s3Wrapper.CreateBucketAsync(_bucketName);
Console.WriteLine($"Successfully created bucket: {_bucketName}");
}
///
/// Create a presigned POST URL.
///
private static async Task CreatePresignedPostAsync()
{
_objectKey = "example-upload.txt";
var expiration = DateTime.UtcNow.AddMinutes(10); // Short expiration for the demo
Console.WriteLine($"Creating presigned POST URL for {_bucketName}/{_objectKey}");
Console.WriteLine($"Expiration: {expiration} UTC");
var s3Client = _s3Wrapper.GetS3Client();
var response = await _s3Wrapper.CreatePresignedPostAsync(
s3Client, _bucketName!, _objectKey, expiration);
Console.WriteLine("Successfully created presigned POST URL");
return response;
}
///
/// Upload a file using the presigned POST URL.
///
private static async Task UploadFileAsync(CreatePresignedPostResponse response)
{
// Create a temporary test file to upload
string testFilePath = Path.GetTempFileName();
string testContent = "This is a test file for the S3 presigned POST scenario.";
await File.WriteAllTextAsync(testFilePath, testContent);
Console.WriteLine($"Created test file at: {testFilePath}");
// Upload the file using the presigned POST URL
Console.WriteLine("\nUploading file using the presigned POST URL...");
var uploadResult = await UploadFileWithPresignedPostAsync(response, testFilePath);
// Display the upload result
if (uploadResult.Success)
{
Console.WriteLine($"Upload successful! Status code: {uploadResult.StatusCode}");
}
else
{
Console.WriteLine($"Upload failed with status code: {uploadResult.StatusCode}");
Console.WriteLine($"Error: {uploadResult.Response}");
throw new Exception("File upload failed");
}
// Clean up the temporary file
File.Delete(testFilePath);
Console.WriteLine("Temporary file deleted");
}
///
/// Helper method to upload a file using a presigned POST URL.
///
private static async Task<(bool Success, HttpStatusCode StatusCode, string Response)> UploadFileWithPresignedPostAsync(
CreatePresignedPostResponse response,
string filePath)
{
try
{
_logger.LogInformation("Uploading file {filePath} using presigned POST URL", filePath);
using var httpClient = _httpClientFactory.CreateClient();
using var formContent = new MultipartFormDataContent();
// Add all the fields from the presigned POST response
foreach (var field in response.Fields)
{
formContent.Add(new StringContent(field.Value), field.Key);
}
// Add the file content
var fileStream = File.OpenRead(filePath);
var fileName = Path.GetFileName(filePath);
var fileContent = new StreamContent(fileStream);
fileContent.Headers.ContentType = new MediaTypeHeaderValue("text/plain");
formContent.Add(fileContent, "file", fileName);
// Send the POST request
var httpResponse = await httpClient.PostAsync(response.Url, formContent);
var responseContent = await httpResponse.Content.ReadAsStringAsync();
// Log and return the result
_logger.LogInformation("Upload completed with status code {statusCode}", httpResponse.StatusCode);
return (httpResponse.IsSuccessStatusCode, httpResponse.StatusCode, responseContent);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error uploading file");
return (false, HttpStatusCode.InternalServerError, ex.Message);
}
}
///
/// Verify that the uploaded file exists in the S3 bucket.
///
private static async Task VerifyFileExistsAsync()
{
_uiMethods.DisplayTitle("Step 5: Verify uploaded file exists");
Console.WriteLine($"Checking if file exists at {_bucketName}/{_objectKey}...");
try
{
var metadata = await _s3Wrapper.GetObjectMetadataAsync(_bucketName!, _objectKey!);
Console.WriteLine($"File verification successful! File exists in the bucket.");
Console.WriteLine($"File size: {metadata.ContentLength} bytes");
Console.WriteLine($"File type: {metadata.Headers.ContentType}");
Console.WriteLine($"Last modified: {metadata.LastModified}");
}
catch (AmazonS3Exception ex) when (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
{
Console.WriteLine($"Error: File was not found in the bucket.");
throw;
}
}
private static void DisplayPresignedPostFields(CreatePresignedPostResponse response)
{
Console.WriteLine($"Presigned POST URL: {response.Url}");
Console.WriteLine("Form fields to include:");
foreach (var field in response.Fields)
{
Console.WriteLine($" {field.Key}: {field.Value}");
}
}
///
/// Clean up resources created by the scenario.
///
private static async Task CleanupAsync()
{
if (!string.IsNullOrEmpty(_bucketName))
{
Console.WriteLine($"Deleting bucket {_bucketName} and its contents...");
bool result = await _s3Wrapper.DeleteBucketAsync(_bucketName);
if (result)
{
Console.WriteLine("Bucket deleted successfully");
}
else
{
Console.WriteLine("Failed to delete bucket - it may have been already deleted");
}
}
}
}
```
------
#### [ C\$1\$1 ]
**SDK for C\$1\$1**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/cpp/example_code/s3#code-examples).
Generate a pre-signed URL to download an object.
```
//! Routine which demonstrates creating a pre-signed URL to download an object from an
//! Amazon Simple Storage Service (Amazon S3) bucket.
/*!
\param bucketName: Name of the bucket.
\param key: Name of an object key.
\param expirationSeconds: Expiration in seconds for pre-signed URL.
\param clientConfig: Aws client configuration.
\return Aws::String: A pre-signed URL.
*/
Aws::String AwsDoc::S3::generatePreSignedGetObjectUrl(const Aws::String &bucketName,
const Aws::String &key,
uint64_t expirationSeconds,
const Aws::S3::S3ClientConfiguration &clientConfig) {
Aws::S3::S3Client client(clientConfig);
return client.GeneratePresignedUrl(bucketName, key, Aws::Http::HttpMethod::HTTP_GET,
expirationSeconds);
}
```
Download using libcurl.
```
static size_t myCurlWriteBack(char *buffer, size_t size, size_t nitems, void *userdata) {
Aws::StringStream *str = (Aws::StringStream *) userdata;
if (nitems > 0) {
str->write(buffer, size * nitems);
}
return size * nitems;
}
//! Utility routine to test getObject with a pre-signed URL.
/*!
\param presignedURL: A pre-signed URL to get an object from a bucket.
\param resultString: A string to hold the result.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::getObjectWithPresignedObjectUrl(const Aws::String &presignedURL,
Aws::String &resultString) {
CURL *curl = curl_easy_init();
CURLcode result;
std::stringstream outWriteString;
result = curl_easy_setopt(curl, CURLOPT_WRITEDATA, &outWriteString);
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_WRITEDATA " << std::endl;
return false;
}
result = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, myCurlWriteBack);
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_WRITEFUNCTION" << std::endl;
return false;
}
result = curl_easy_setopt(curl, CURLOPT_URL, presignedURL.c_str());
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_URL" << std::endl;
return false;
}
result = curl_easy_perform(curl);
if (result != CURLE_OK) {
std::cerr << "Failed to perform CURL request" << std::endl;
return false;
}
resultString = outWriteString.str();
if (resultString.find("read(buffer, size * nitems);
return str->gcount();
}
static size_t myCurlWriteBack(char *buffer, size_t size, size_t nitems, void *userdata) {
Aws::StringStream *str = (Aws::StringStream *) userdata;
if (nitems > 0) {
str->write(buffer, size * nitems);
}
return size * nitems;
}
//! Utility routine to test putObject with a pre-signed URL.
/*!
\param presignedURL: A pre-signed URL to put an object in a bucket.
\param data: Body of the putObject request.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::PutStringWithPresignedObjectURL(const Aws::String &presignedURL,
const Aws::String &data) {
CURL *curl = curl_easy_init();
CURLcode result;
Aws::StringStream readStringStream;
readStringStream << data;
result = curl_easy_setopt(curl, CURLOPT_READFUNCTION, myCurlReadBack);
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_READFUNCTION" << std::endl;
return false;
}
result = curl_easy_setopt(curl, CURLOPT_READDATA, &readStringStream);
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_READDATA" << std::endl;
return false;
}
result = curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
(curl_off_t) data.size());
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_INFILESIZE_LARGE" << std::endl;
return false;
}
result = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, myCurlWriteBack);
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_WRITEFUNCTION" << std::endl;
return false;
}
std::stringstream outWriteString;
result = curl_easy_setopt(curl, CURLOPT_WRITEDATA, &outWriteString);
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_WRITEDATA " << std::endl;
return false;
}
result = curl_easy_setopt(curl, CURLOPT_URL, presignedURL.c_str());
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_URL" << std::endl;
return false;
}
result = curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
if (result != CURLE_OK) {
std::cerr << "Failed to set CURLOPT_PUT" << std::endl;
return false;
}
result = curl_easy_perform(curl);
if (result != CURLE_OK) {
std::cerr << "Failed to perform CURL request" << std::endl;
return false;
}
std::string outString = outWriteString.str();
if (outString.empty()) {
std::cout << "Successfully put object." << std::endl;
return true;
} else {
std::cout << "A server error was encountered, output:\n" << outString
<< std::endl;
return false;
}
}
```
------
#### [ Go ]
**SDK for Go V2**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/gov2/s3#code-examples).
Create functions that wrap S3 presigning actions.
```
import (
"context"
"log"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
// Presigner encapsulates the Amazon Simple Storage Service (Amazon S3) presign actions
// used in the examples.
// It contains PresignClient, a client that is used to presign requests to Amazon S3.
// Presigned requests contain temporary credentials and can be made from any HTTP client.
type Presigner struct {
PresignClient *s3.PresignClient
}
// GetObject makes a presigned request that can be used to get an object from a bucket.
// The presigned request is valid for the specified number of seconds.
func (presigner Presigner) GetObject(
ctx context.Context, bucketName string, objectKey string, lifetimeSecs int64) (*v4.PresignedHTTPRequest, error) {
request, err := presigner.PresignClient.PresignGetObject(ctx, &s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
}, func(opts *s3.PresignOptions) {
opts.Expires = time.Duration(lifetimeSecs * int64(time.Second))
})
if err != nil {
log.Printf("Couldn't get a presigned request to get %v:%v. Here's why: %v\n",
bucketName, objectKey, err)
}
return request, err
}
// PutObject makes a presigned request that can be used to put an object in a bucket.
// The presigned request is valid for the specified number of seconds.
func (presigner Presigner) PutObject(
ctx context.Context, bucketName string, objectKey string, lifetimeSecs int64) (*v4.PresignedHTTPRequest, error) {
request, err := presigner.PresignClient.PresignPutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
}, func(opts *s3.PresignOptions) {
opts.Expires = time.Duration(lifetimeSecs * int64(time.Second))
})
if err != nil {
log.Printf("Couldn't get a presigned request to put %v:%v. Here's why: %v\n",
bucketName, objectKey, err)
}
return request, err
}
// DeleteObject makes a presigned request that can be used to delete an object from a bucket.
func (presigner Presigner) DeleteObject(ctx context.Context, bucketName string, objectKey string) (*v4.PresignedHTTPRequest, error) {
request, err := presigner.PresignClient.PresignDeleteObject(ctx, &s3.DeleteObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
if err != nil {
log.Printf("Couldn't get a presigned request to delete object %v. Here's why: %v\n", objectKey, err)
}
return request, err
}
func (presigner Presigner) PresignPostObject(ctx context.Context, bucketName string, objectKey string, lifetimeSecs int64) (*s3.PresignedPostRequest, error) {
request, err := presigner.PresignClient.PresignPostObject(ctx, &s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
}, func(options *s3.PresignPostOptions) {
options.Expires = time.Duration(lifetimeSecs) * time.Second
})
if err != nil {
log.Printf("Couldn't get a presigned post request to put %v:%v. Here's why: %v\n", bucketName, objectKey, err)
}
return request, nil
}
```
Run an interactive example that generates and uses presigned URLs to upload, download, and delete an S3 object.
```
import (
"bytes"
"context"
"io"
"log"
"mime/multipart"
"net/http"
"os"
"strings"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/awsdocs/aws-doc-sdk-examples/gov2/demotools"
"github.com/awsdocs/aws-doc-sdk-examples/gov2/s3/actions"
)
// RunPresigningScenario is an interactive example that shows you how to get presigned
// HTTP requests that you can use to move data into and out of Amazon Simple Storage
// Service (Amazon S3). The presigned requests contain temporary credentials and can
// be used by an HTTP client.
//
// 1. Get a presigned request to put an object in a bucket.
// 2. Use the net/http package to use the presigned request to upload a local file to the bucket.
// 3. Get a presigned request to get an object from a bucket.
// 4. Use the net/http package to use the presigned request to download the object to a local file.
// 5. Get a presigned request to delete an object from a bucket.
// 6. Use the net/http package to use the presigned request to delete the object.
//
// This example creates an Amazon S3 presign client from the specified sdkConfig so that
// you can replace it with a mocked or stubbed config for unit testing.
//
// It uses a questioner from the `demotools` package to get input during the example.
// This package can be found in the ..\..\demotools folder of this repo.
//
// It uses an IHttpRequester interface to abstract HTTP requests so they can be mocked
// during testing.
func RunPresigningScenario(ctx context.Context, sdkConfig aws.Config, questioner demotools.IQuestioner, httpRequester IHttpRequester) {
defer func() {
if r := recover(); r != nil {
log.Println("Something went wrong with the demo.")
_, isMock := questioner.(*demotools.MockQuestioner)
if isMock || questioner.AskBool("Do you want to see the full error message (y/n)?", "y") {
log.Println(r)
}
}
}()
log.Println(strings.Repeat("-", 88))
log.Println("Welcome to the Amazon S3 presigning demo.")
log.Println(strings.Repeat("-", 88))
s3Client := s3.NewFromConfig(sdkConfig)
bucketBasics := actions.BucketBasics{S3Client: s3Client}
presignClient := s3.NewPresignClient(s3Client)
presigner := actions.Presigner{PresignClient: presignClient}
bucketName := questioner.Ask("We'll need a bucket. Enter a name for a bucket "+
"you own or one you want to create:", demotools.NotEmpty{})
bucketExists, err := bucketBasics.BucketExists(ctx, bucketName)
if err != nil {
panic(err)
}
if !bucketExists {
err = bucketBasics.CreateBucket(ctx, bucketName, sdkConfig.Region)
if err != nil {
panic(err)
} else {
log.Println("Bucket created.")
}
}
log.Println(strings.Repeat("-", 88))
log.Printf("Let's presign a request to upload a file to your bucket.")
uploadFilename := questioner.Ask("Enter the path to a file you want to upload:",
demotools.NotEmpty{})
uploadKey := questioner.Ask("What would you like to name the uploaded object?",
demotools.NotEmpty{})
uploadFile, err := os.Open(uploadFilename)
if err != nil {
panic(err)
}
defer uploadFile.Close()
presignedPutRequest, err := presigner.PutObject(ctx, bucketName, uploadKey, 60)
if err != nil {
panic(err)
}
log.Printf("Got a presigned %v request to URL:\n\t%v\n", presignedPutRequest.Method,
presignedPutRequest.URL)
log.Println("Using net/http to send the request...")
info, err := uploadFile.Stat()
if err != nil {
panic(err)
}
putResponse, err := httpRequester.Put(presignedPutRequest.URL, info.Size(), uploadFile)
if err != nil {
panic(err)
}
log.Printf("%v object %v with presigned URL returned %v.", presignedPutRequest.Method,
uploadKey, putResponse.StatusCode)
log.Println(strings.Repeat("-", 88))
log.Printf("Let's presign a request to download the object.")
questioner.Ask("Press Enter when you're ready.")
presignedGetRequest, err := presigner.GetObject(ctx, bucketName, uploadKey, 60)
if err != nil {
panic(err)
}
log.Printf("Got a presigned %v request to URL:\n\t%v\n", presignedGetRequest.Method,
presignedGetRequest.URL)
log.Println("Using net/http to send the request...")
getResponse, err := httpRequester.Get(presignedGetRequest.URL)
if err != nil {
panic(err)
}
log.Printf("%v object %v with presigned URL returned %v.", presignedGetRequest.Method,
uploadKey, getResponse.StatusCode)
defer getResponse.Body.Close()
downloadBody, err := io.ReadAll(getResponse.Body)
if err != nil {
panic(err)
}
log.Printf("Downloaded %v bytes. Here are the first 100 of them:\n", len(downloadBody))
log.Println(strings.Repeat("-", 88))
log.Println(string(downloadBody[:100]))
log.Println(strings.Repeat("-", 88))
log.Println("Now we'll create a new request to put the same object using a presigned post request")
questioner.Ask("Press Enter when you're ready.")
presignPostRequest, err := presigner.PresignPostObject(ctx, bucketName, uploadKey, 60)
if err != nil {
panic(err)
}
log.Printf("Got a presigned post request to url %v with values %v\n", presignPostRequest.URL, presignPostRequest.Values)
log.Println("Using net/http multipart to send the request...")
uploadFile, err = os.Open(uploadFilename)
if err != nil {
panic(err)
}
defer uploadFile.Close()
multiPartResponse, err := sendMultipartRequest(presignPostRequest.URL, presignPostRequest.Values, uploadFile, uploadKey, httpRequester)
if err != nil {
panic(err)
}
log.Printf("Presign post object %v with presigned URL returned %v.", uploadKey, multiPartResponse.StatusCode)
log.Println("Let's presign a request to delete the object.")
questioner.Ask("Press Enter when you're ready.")
presignedDelRequest, err := presigner.DeleteObject(ctx, bucketName, uploadKey)
if err != nil {
panic(err)
}
log.Printf("Got a presigned %v request to URL:\n\t%v\n", presignedDelRequest.Method,
presignedDelRequest.URL)
log.Println("Using net/http to send the request...")
delResponse, err := httpRequester.Delete(presignedDelRequest.URL)
if err != nil {
panic(err)
}
log.Printf("%v object %v with presigned URL returned %v.\n", presignedDelRequest.Method,
uploadKey, delResponse.StatusCode)
log.Println(strings.Repeat("-", 88))
log.Println("Thanks for watching!")
log.Println(strings.Repeat("-", 88))
}
```
Define an HTTP request wrapper used by the example to make HTTP requests.
```
// IHttpRequester abstracts HTTP requests into an interface so it can be mocked during
// unit testing.
type IHttpRequester interface {
Get(url string) (resp *http.Response, err error)
Post(url, contentType string, body io.Reader) (resp *http.Response, err error)
Put(url string, contentLength int64, body io.Reader) (resp *http.Response, err error)
Delete(url string) (resp *http.Response, err error)
}
// HttpRequester uses the net/http package to make HTTP requests during the scenario.
type HttpRequester struct{}
func (httpReq HttpRequester) Get(url string) (resp *http.Response, err error) {
return http.Get(url)
}
func (httpReq HttpRequester) Post(url, contentType string, body io.Reader) (resp *http.Response, err error) {
postRequest, err := http.NewRequest("POST", url, body)
if err != nil {
return nil, err
}
postRequest.Header.Set("Content-Type", contentType)
return http.DefaultClient.Do(postRequest)
}
func (httpReq HttpRequester) Put(url string, contentLength int64, body io.Reader) (resp *http.Response, err error) {
putRequest, err := http.NewRequest("PUT", url, body)
if err != nil {
return nil, err
}
putRequest.ContentLength = contentLength
return http.DefaultClient.Do(putRequest)
}
func (httpReq HttpRequester) Delete(url string) (resp *http.Response, err error) {
delRequest, err := http.NewRequest("DELETE", url, nil)
if err != nil {
return nil, err
}
return http.DefaultClient.Do(delRequest)
}
```
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
The following shows three example of how to create presigned URLs and use the URLs with HTTP client libraries:
+ An HTTP GET request that uses the URL with three HTTP client libraries
+ An HTTP PUT request with metadata in headers that uses the URL with three HTTP client libraries
+ An HTTP PUT request with query parameters that uses the URL with one HTTP client library
Generate a pre-signed URL for an object, then download it (GET request).
Imports.
```
import com.example.s3.util.PresignUrlUtils;
import org.slf4j.Logger;
import software.amazon.awssdk.http.HttpExecuteRequest;
import software.amazon.awssdk.http.HttpExecuteResponse;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.SdkHttpMethod;
import software.amazon.awssdk.http.SdkHttpRequest;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
import software.amazon.awssdk.utils.IoUtils;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URISyntaxException;
import java.net.URL;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Paths;
import java.time.Duration;
import java.util.UUID;
```
Generate the URL.
```
/* Create a pre-signed URL to download an object in a subsequent GET request. */
public String createPresignedGetUrl(String bucketName, String keyName) {
try (S3Presigner presigner = S3Presigner.create()) {
GetObjectRequest objectRequest = GetObjectRequest.builder()
.bucket(bucketName)
.key(keyName)
.build();
GetObjectPresignRequest presignRequest = GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(10)) // The URL will expire in 10 minutes.
.getObjectRequest(objectRequest)
.build();
PresignedGetObjectRequest presignedRequest = presigner.presignGetObject(presignRequest);
logger.info("Presigned URL: [{}]", presignedRequest.url().toString());
logger.info("HTTP method: [{}]", presignedRequest.httpRequest().method());
return presignedRequest.url().toExternalForm();
}
}
```
Download the object by using any one of the following three approaches.
Use JDK `HttpURLConnection` (since v1.1) class to do the download.
```
/* Use the JDK HttpURLConnection (since v1.1) class to do the download. */
public byte[] useHttpUrlConnectionToGet(String presignedUrlString) {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); // Capture the response body to a byte array.
try {
URL presignedUrl = new URL(presignedUrlString);
HttpURLConnection connection = (HttpURLConnection) presignedUrl.openConnection();
connection.setRequestMethod("GET");
// Download the result of executing the request.
try (InputStream content = connection.getInputStream()) {
IoUtils.copy(content, byteArrayOutputStream);
}
logger.info("HTTP response code is " + connection.getResponseCode());
} catch (S3Exception | IOException e) {
logger.error(e.getMessage(), e);
}
return byteArrayOutputStream.toByteArray();
}
```
Use JDK `HttpClient` (since v11) class to do the download.
```
/* Use the JDK HttpClient (since v11) class to do the download. */
public byte[] useHttpClientToGet(String presignedUrlString) {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); // Capture the response body to a byte array.
HttpRequest.Builder requestBuilder = HttpRequest.newBuilder();
HttpClient httpClient = HttpClient.newHttpClient();
try {
URL presignedUrl = new URL(presignedUrlString);
HttpResponse response = httpClient.send(requestBuilder
.uri(presignedUrl.toURI())
.GET()
.build(),
HttpResponse.BodyHandlers.ofInputStream());
IoUtils.copy(response.body(), byteArrayOutputStream);
logger.info("HTTP response code is " + response.statusCode());
} catch (URISyntaxException | InterruptedException | IOException e) {
logger.error(e.getMessage(), e);
}
return byteArrayOutputStream.toByteArray();
}
```
Use the AWS SDK for Java `SdkHttpClient` class to do the download.
```
/* Use the AWS SDK for Java SdkHttpClient class to do the download. */
public byte[] useSdkHttpClientToGet(String presignedUrlString) {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); // Capture the response body to a byte array.
try {
URL presignedUrl = new URL(presignedUrlString);
SdkHttpRequest request = SdkHttpRequest.builder()
.method(SdkHttpMethod.GET)
.uri(presignedUrl.toURI())
.build();
HttpExecuteRequest executeRequest = HttpExecuteRequest.builder()
.request(request)
.build();
try (SdkHttpClient sdkHttpClient = ApacheHttpClient.create()) {
HttpExecuteResponse response = sdkHttpClient.prepareRequest(executeRequest).call();
response.responseBody().ifPresentOrElse(
abortableInputStream -> {
try {
IoUtils.copy(abortableInputStream, byteArrayOutputStream);
} catch (IOException e) {
throw new RuntimeException(e);
}
},
() -> logger.error("No response body."));
logger.info("HTTP Response code is {}", response.httpResponse().statusCode());
}
} catch (URISyntaxException | IOException e) {
logger.error(e.getMessage(), e);
}
return byteArrayOutputStream.toByteArray();
}
```
Generate a pre-signed URL with metadata in headers for an upload, then upload a file (PUT request).
Imports.
```
import com.example.s3.util.PresignUrlUtils;
import org.slf4j.Logger;
import software.amazon.awssdk.core.internal.sync.FileContentStreamProvider;
import software.amazon.awssdk.http.HttpExecuteRequest;
import software.amazon.awssdk.http.HttpExecuteResponse;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.SdkHttpMethod;
import software.amazon.awssdk.http.SdkHttpRequest;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest;
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.io.RandomAccessFile;
import java.net.HttpURLConnection;
import java.net.URISyntaxException;
import java.net.URL;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Duration;
import java.util.Map;
import java.util.UUID;
```
Generate the URL.
```
/* Create a presigned URL to use in a subsequent PUT request */
public String createPresignedUrl(String bucketName, String keyName, Map metadata) {
try (S3Presigner presigner = S3Presigner.create()) {
PutObjectRequest objectRequest = PutObjectRequest.builder()
.bucket(bucketName)
.key(keyName)
.metadata(metadata)
.build();
PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(10)) // The URL expires in 10 minutes.
.putObjectRequest(objectRequest)
.build();
PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest);
String myURL = presignedRequest.url().toString();
logger.info("Presigned URL to upload a file to: [{}]", myURL);
logger.info("HTTP method: [{}]", presignedRequest.httpRequest().method());
return presignedRequest.url().toExternalForm();
}
}
```
Upload a file object by using any one of the following three approaches.
Use the JDK `HttpURLConnection` (since v1.1) class to do the upload.
```
/* Use the JDK HttpURLConnection (since v1.1) class to do the upload. */
public void useHttpUrlConnectionToPut(String presignedUrlString, File fileToPut, Map metadata) {
logger.info("Begin [{}] upload", fileToPut.toString());
try {
URL presignedUrl = new URL(presignedUrlString);
HttpURLConnection connection = (HttpURLConnection) presignedUrl.openConnection();
connection.setDoOutput(true);
metadata.forEach((k, v) -> connection.setRequestProperty("x-amz-meta-" + k, v));
connection.setRequestMethod("PUT");
OutputStream out = connection.getOutputStream();
try (RandomAccessFile file = new RandomAccessFile(fileToPut, "r");
FileChannel inChannel = file.getChannel()) {
ByteBuffer buffer = ByteBuffer.allocate(8192); //Buffer size is 8k
while (inChannel.read(buffer) > 0) {
buffer.flip();
for (int i = 0; i < buffer.limit(); i++) {
out.write(buffer.get());
}
buffer.clear();
}
} catch (IOException e) {
logger.error(e.getMessage(), e);
}
out.close();
connection.getResponseCode();
logger.info("HTTP response code is " + connection.getResponseCode());
} catch (S3Exception | IOException e) {
logger.error(e.getMessage(), e);
}
}
```
Use the JDK `HttpClient` (since v11) class to do the upload.
```
/* Use the JDK HttpClient (since v11) class to do the upload. */
public void useHttpClientToPut(String presignedUrlString, File fileToPut, Map metadata) {
logger.info("Begin [{}] upload", fileToPut.toString());
HttpRequest.Builder requestBuilder = HttpRequest.newBuilder();
metadata.forEach((k, v) -> requestBuilder.header("x-amz-meta-" + k, v));
HttpClient httpClient = HttpClient.newHttpClient();
try {
final HttpResponse response = httpClient.send(requestBuilder
.uri(new URL(presignedUrlString).toURI())
.PUT(HttpRequest.BodyPublishers.ofFile(Path.of(fileToPut.toURI())))
.build(),
HttpResponse.BodyHandlers.discarding());
logger.info("HTTP response code is " + response.statusCode());
} catch (URISyntaxException | InterruptedException | IOException e) {
logger.error(e.getMessage(), e);
}
}
```
Use the AWS for Java V2 `SdkHttpClient` class to do the upload.
```
/* Use the AWS SDK for Java V2 SdkHttpClient class to do the upload. */
public void useSdkHttpClientToPut(String presignedUrlString, File fileToPut, Map metadata) {
logger.info("Begin [{}] upload", fileToPut.toString());
try {
URL presignedUrl = new URL(presignedUrlString);
SdkHttpRequest.Builder requestBuilder = SdkHttpRequest.builder()
.method(SdkHttpMethod.PUT)
.uri(presignedUrl.toURI());
// Add headers
metadata.forEach((k, v) -> requestBuilder.putHeader("x-amz-meta-" + k, v));
// Finish building the request.
SdkHttpRequest request = requestBuilder.build();
HttpExecuteRequest executeRequest = HttpExecuteRequest.builder()
.request(request)
.contentStreamProvider(new FileContentStreamProvider(fileToPut.toPath()))
.build();
try (SdkHttpClient sdkHttpClient = ApacheHttpClient.create()) {
HttpExecuteResponse response = sdkHttpClient.prepareRequest(executeRequest).call();
logger.info("Response code: {}", response.httpResponse().statusCode());
}
} catch (URISyntaxException | IOException e) {
logger.error(e.getMessage(), e);
}
}
```
Generate a pre-signed URL with query parameters for an upload, then upload a file (PUT request).
Imports.
```
import com.example.s3.util.PresignUrlUtils;
import org.slf4j.Logger;
import software.amazon.awssdk.awscore.AwsRequestOverrideConfiguration;
import software.amazon.awssdk.core.internal.sync.FileContentStreamProvider;
import software.amazon.awssdk.http.HttpExecuteRequest;
import software.amazon.awssdk.http.HttpExecuteResponse;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.SdkHttpMethod;
import software.amazon.awssdk.http.SdkHttpRequest;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest;
import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.file.Paths;
import java.time.Duration;
import java.util.Map;
import java.util.UUID;
```
Generate the URL.
```
/**
* Creates a presigned URL to use in a subsequent HTTP PUT request. The code adds query parameters
* to the request instead of using headers. By using query parameters, you do not need to add the
* the parameters as headers when the PUT request is eventually sent.
*
* @param bucketName Bucket name where the object will be uploaded.
* @param keyName Key name of the object that will be uploaded.
* @param queryParams Query string parameters to be added to the presigned URL.
* @return
*/
public String createPresignedUrl(String bucketName, String keyName, Map queryParams) {
try (S3Presigner presigner = S3Presigner.create()) {
// Create an override configuration to store the query parameters.
AwsRequestOverrideConfiguration.Builder overrideConfigurationBuilder = AwsRequestOverrideConfiguration.builder();
queryParams.forEach(overrideConfigurationBuilder::putRawQueryParameter);
PutObjectRequest objectRequest = PutObjectRequest.builder()
.bucket(bucketName)
.key(keyName)
.overrideConfiguration(overrideConfigurationBuilder.build()) // Add the override configuration.
.build();
PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(10)) // The URL expires in 10 minutes.
.putObjectRequest(objectRequest)
.build();
PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest);
String myURL = presignedRequest.url().toString();
logger.info("Presigned URL to upload a file to: [{}]", myURL);
logger.info("HTTP method: [{}]", presignedRequest.httpRequest().method());
return presignedRequest.url().toExternalForm();
}
}
```
Use the AWS for Java V2 `SdkHttpClient` class to do the upload.
```
/**
* Use the AWS SDK for Java V2 SdkHttpClient class to execute the PUT request. Since the
* URL contains the query parameters, no headers are needed for metadata, SSE settings, or ACL settings.
*
* @param presignedUrlString The URL for the PUT request.
* @param fileToPut File to uplaod
*/
public void useSdkHttpClientToPut(String presignedUrlString, File fileToPut) {
logger.info("Begin [{}] upload", fileToPut.toString());
try {
URL presignedUrl = new URL(presignedUrlString);
SdkHttpRequest.Builder requestBuilder = SdkHttpRequest.builder()
.method(SdkHttpMethod.PUT)
.uri(presignedUrl.toURI());
SdkHttpRequest request = requestBuilder.build();
HttpExecuteRequest executeRequest = HttpExecuteRequest.builder()
.request(request)
.contentStreamProvider(new FileContentStreamProvider(fileToPut.toPath()))
.build();
try (SdkHttpClient sdkHttpClient = ApacheHttpClient.create()) {
HttpExecuteResponse response = sdkHttpClient.prepareRequest(executeRequest).call();
logger.info("Response code: {}", response.httpResponse().statusCode());
}
} catch (URISyntaxException | IOException e) {
logger.error(e.getMessage(), e);
}
}
```
------
#### [ JavaScript ]
**SDK for JavaScript (v3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javascriptv3/example_code/s3#code-examples).
Create a presigned URL to upload an object to a bucket.
```
import https from "node:https";
import { XMLParser } from "fast-xml-parser";
import { PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { fromIni } from "@aws-sdk/credential-providers";
import { HttpRequest } from "@smithy/protocol-http";
import {
getSignedUrl,
S3RequestPresigner,
} from "@aws-sdk/s3-request-presigner";
import { parseUrl } from "@smithy/url-parser";
import { formatUrl } from "@aws-sdk/util-format-url";
import { Hash } from "@smithy/hash-node";
const createPresignedUrlWithoutClient = async ({ region, bucket, key }) => {
const url = parseUrl(`https://${bucket}.s3.${region}.amazonaws.com/${key}`);
const presigner = new S3RequestPresigner({
credentials: fromIni(),
region,
sha256: Hash.bind(null, "sha256"),
});
const signedUrlObject = await presigner.presign(
new HttpRequest({ ...url, method: "PUT" }),
);
return formatUrl(signedUrlObject);
};
const createPresignedUrlWithClient = ({ region, bucket, key }) => {
const client = new S3Client({ region });
const command = new PutObjectCommand({ Bucket: bucket, Key: key });
return getSignedUrl(client, command, { expiresIn: 3600 });
};
/**
* Make a PUT request to the provided URL.
*
* @param {string} url
* @param {string} data
*/
const put = (url, data) => {
return new Promise((resolve, reject) => {
const req = https.request(
url,
{ method: "PUT", headers: { "Content-Length": new Blob([data]).size } },
(res) => {
let responseBody = "";
res.on("data", (chunk) => {
responseBody += chunk;
});
res.on("end", () => {
const parser = new XMLParser();
if (res.statusCode >= 200 && res.statusCode <= 299) {
resolve(parser.parse(responseBody, true));
} else {
reject(parser.parse(responseBody, true));
}
});
},
);
req.on("error", (err) => {
reject(err);
});
req.write(data);
req.end();
});
};
/**
* Create two presigned urls for uploading an object to an S3 bucket.
* The first presigned URL is created with credentials from the shared INI file
* in the current environment. The second presigned URL is created using an
* existing S3Client instance that has already been provided with credentials.
* @param {{ bucketName: string, key: string, region: string }}
*/
export const main = async ({ bucketName, key, region }) => {
try {
const noClientUrl = await createPresignedUrlWithoutClient({
bucket: bucketName,
key,
region,
});
const clientUrl = await createPresignedUrlWithClient({
bucket: bucketName,
region,
key,
});
// After you get the presigned URL, you can provide your own file
// data. Refer to put() above.
console.log("Calling PUT using presigned URL without client");
await put(noClientUrl, "Hello World");
console.log("Calling PUT using presigned URL with client");
await put(clientUrl, "Hello World");
console.log("\nDone. Check your S3 console.");
} catch (caught) {
if (caught instanceof Error && caught.name === "CredentialsProviderError") {
console.error(
`There was an error getting your credentials. Are your local credentials configured?\n${caught.name}: ${caught.message}`,
);
} else {
throw caught;
}
}
};
```
Create a presigned URL to download an object from a bucket.
```
import { GetObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { fromIni } from "@aws-sdk/credential-providers";
import { HttpRequest } from "@smithy/protocol-http";
import {
getSignedUrl,
S3RequestPresigner,
} from "@aws-sdk/s3-request-presigner";
import { parseUrl } from "@smithy/url-parser";
import { formatUrl } from "@aws-sdk/util-format-url";
import { Hash } from "@smithy/hash-node";
const createPresignedUrlWithoutClient = async ({ region, bucket, key }) => {
const url = parseUrl(`https://${bucket}.s3.${region}.amazonaws.com/${key}`);
const presigner = new S3RequestPresigner({
credentials: fromIni(),
region,
sha256: Hash.bind(null, "sha256"),
});
const signedUrlObject = await presigner.presign(new HttpRequest(url));
return formatUrl(signedUrlObject);
};
const createPresignedUrlWithClient = ({ region, bucket, key }) => {
const client = new S3Client({ region });
const command = new GetObjectCommand({ Bucket: bucket, Key: key });
return getSignedUrl(client, command, { expiresIn: 3600 });
};
/**
* Create two presigned urls for downloading an object from an S3 bucket.
* The first presigned URL is created with credentials from the shared INI file
* in the current environment. The second presigned URL is created using an
* existing S3Client instance that has already been provided with credentials.
* @param {{ bucketName: string, key: string, region: string }}
*/
export const main = async ({ bucketName, key, region }) => {
try {
const noClientUrl = await createPresignedUrlWithoutClient({
bucket: bucketName,
region,
key,
});
const clientUrl = await createPresignedUrlWithClient({
bucket: bucketName,
region,
key,
});
console.log("Presigned URL without client");
console.log(noClientUrl);
console.log("\n");
console.log("Presigned URL with client");
console.log(clientUrl);
} catch (caught) {
if (caught instanceof Error && caught.name === "CredentialsProviderError") {
console.error(
`There was an error getting your credentials. Are your local credentials configured?\n${caught.name}: ${caught.message}`,
);
} else {
throw caught;
}
}
};
```
+ For more information, see [AWS SDK for JavaScript Developer Guide](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/s3-example-creating-buckets.html#s3-create-presigendurl).
------
#### [ Kotlin ]
**SDK for Kotlin**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/kotlin/services/s3#code-examples).
Create a `GetObject` presigned request and use the URL to download an object.
```
suspend fun getObjectPresigned(
s3: S3Client,
bucketName: String,
keyName: String,
): String {
// Create a GetObjectRequest.
val unsignedRequest =
GetObjectRequest {
bucket = bucketName
key = keyName
}
// Presign the GetObject request.
val presignedRequest = s3.presignGetObject(unsignedRequest, 24.hours)
// Use the URL from the presigned HttpRequest in a subsequent HTTP GET request to retrieve the object.
val objectContents = URL(presignedRequest.url.toString()).readText()
return objectContents
}
```
Create a `GetObject` presigned request with advanced options.
```
suspend fun getObjectPresignedMoreOptions(
s3: S3Client,
bucketName: String,
keyName: String,
): HttpRequest {
// Create a GetObjectRequest.
val unsignedRequest =
GetObjectRequest {
bucket = bucketName
key = keyName
}
// Presign the GetObject request.
val presignedRequest =
s3.presignGetObject(unsignedRequest, signer = CrtAwsSigner) {
signingDate = Instant.now() + 12.hours // Presigned request can be used 12 hours from now.
algorithm = AwsSigningAlgorithm.SIGV4_ASYMMETRIC
signatureType = AwsSignatureType.HTTP_REQUEST_VIA_QUERY_PARAMS
expiresAfter = 8.hours // Presigned request expires 8 hours later.
}
return presignedRequest
}
```
Create a `PutObject` presigned request and use it to upload an object.
```
suspend fun putObjectPresigned(
s3: S3Client,
bucketName: String,
keyName: String,
content: String,
) {
// Create a PutObjectRequest.
val unsignedRequest =
PutObjectRequest {
bucket = bucketName
key = keyName
}
// Presign the request.
val presignedRequest = s3.presignPutObject(unsignedRequest, 24.hours)
// Use the URL and any headers from the presigned HttpRequest in a subsequent HTTP PUT request to retrieve the object.
// Create a PUT request using the OKHttpClient API.
val putRequest =
Request
.Builder()
.url(presignedRequest.url.toString())
.apply {
presignedRequest.headers.forEach { key, values ->
header(key, values.joinToString(", "))
}
}.put(content.toRequestBody())
.build()
val response = OkHttpClient().newCall(putRequest).execute()
assert(response.isSuccessful)
}
```
+ For more information, see [AWS SDK for Kotlin developer guide](https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/presign-requests.html).
------
#### [ PHP ]
**SDK for PHP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/php/example_code/s3#code-examples).
```
namespace S3;
use Aws\Exception\AwsException;
use AwsUtilities\PrintableLineBreak;
use AwsUtilities\TestableReadline;
use DateTime;
require 'vendor/autoload.php';
class PresignedURL
{
use PrintableLineBreak;
use TestableReadline;
public function run()
{
$s3Service = new S3Service();
$expiration = new DateTime("+20 minutes");
$linebreak = $this->getLineBreak();
echo $linebreak;
echo ("Welcome to the Amazon S3 presigned URL demo.\n");
echo $linebreak;
$bucket = $this->testable_readline("First, please enter the name of the S3 bucket to use: ");
$key = $this->testable_readline("Next, provide the key of an object in the given bucket: ");
echo $linebreak;
$command = $s3Service->getClient()->getCommand('GetObject', [
'Bucket' => $bucket,
'Key' => $key,
]);
try {
$preSignedUrl = $s3Service->preSignedUrl($command, $expiration);
echo "Your preSignedUrl is \n$preSignedUrl\nand will be good for the next 20 minutes.\n";
echo $linebreak;
echo "Thanks for trying the Amazon S3 presigned URL demo.\n";
} catch (AwsException $exception) {
echo $linebreak;
echo "Something went wrong: $exception";
die();
}
}
}
$runner = new PresignedURL();
$runner->run();
namespace S3;
use Aws\CommandInterface;
use Aws\Exception\AwsException;
use Aws\Result;
use Aws\S3\Exception\S3Exception;
use Aws\S3\S3Client;
use AwsUtilities\AWSServiceClass;
use DateTimeInterface;
class S3Service extends AWSServiceClass
{
protected S3Client $client;
protected bool $verbose;
public function __construct(S3Client $client = null, $verbose = false)
{
if ($client) {
$this->client = $client;
} else {
$this->client = new S3Client([
'version' => 'latest',
'region' => 'us-west-2',
]);
}
$this->verbose = $verbose;
}
public function setVerbose($verbose)
{
$this->verbose = $verbose;
}
public function isVerbose(): bool
{
return $this->verbose;
}
public function getClient(): S3Client
{
return $this->client;
}
public function setClient(S3Client $client)
{
$this->client = $client;
}
public function emptyAndDeleteBucket($bucketName, array $args = [])
{
try {
$objects = $this->listAllObjects($bucketName, $args);
$this->deleteObjects($bucketName, $objects, $args);
if ($this->verbose) {
echo "Deleted all objects and folders from $bucketName.\n";
}
$this->deleteBucket($bucketName, $args);
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to delete $bucketName with error: {$exception->getMessage()}\n";
echo "\nPlease fix error with bucket deletion before continuing.\n";
}
throw $exception;
}
}
public function createBucket(string $bucketName, array $args = [])
{
$parameters = array_merge(['Bucket' => $bucketName], $args);
try {
$this->client->createBucket($parameters);
if ($this->verbose) {
echo "Created the bucket named: $bucketName.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to create $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with bucket creation before continuing.";
}
throw $exception;
}
}
public function putObject(string $bucketName, string $key, array $args = [])
{
$parameters = array_merge(['Bucket' => $bucketName, 'Key' => $key], $args);
try {
$this->client->putObject($parameters);
if ($this->verbose) {
echo "Uploaded the object named: $key to the bucket named: $bucketName.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to create $key in $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with object uploading before continuing.";
}
throw $exception;
}
}
public function getObject(string $bucketName, string $key, array $args = []): Result
{
$parameters = array_merge(['Bucket' => $bucketName, 'Key' => $key], $args);
try {
$object = $this->client->getObject($parameters);
if ($this->verbose) {
echo "Downloaded the object named: $key to the bucket named: $bucketName.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to download $key from $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with object downloading before continuing.";
}
throw $exception;
}
return $object;
}
public function copyObject($bucketName, $key, $copySource, array $args = [])
{
$parameters = array_merge(['Bucket' => $bucketName, 'Key' => $key, "CopySource" => $copySource], $args);
try {
$this->client->copyObject($parameters);
if ($this->verbose) {
echo "Copied the object from: $copySource in $bucketName to: $key.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to copy $copySource in $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with object copying before continuing.";
}
throw $exception;
}
}
public function listObjects(string $bucketName, $start = 0, $max = 1000, array $args = [])
{
$parameters = array_merge(['Bucket' => $bucketName, 'Marker' => $start, "MaxKeys" => $max], $args);
try {
$objects = $this->client->listObjectsV2($parameters);
if ($this->verbose) {
echo "Retrieved the list of objects from: $bucketName.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to retrieve the objects from $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with list objects before continuing.";
}
throw $exception;
}
return $objects;
}
public function listAllObjects($bucketName, array $args = [])
{
$parameters = array_merge(['Bucket' => $bucketName], $args);
$contents = [];
$paginator = $this->client->getPaginator("ListObjectsV2", $parameters);
foreach ($paginator as $result) {
if($result['KeyCount'] == 0){
break;
}
foreach ($result['Contents'] as $object) {
$contents[] = $object;
}
}
return $contents;
}
public function deleteObjects(string $bucketName, array $objects, array $args = [])
{
$listOfObjects = array_map(
function ($object) {
return ['Key' => $object];
},
array_column($objects, 'Key')
);
if(!$listOfObjects){
return;
}
$parameters = array_merge(['Bucket' => $bucketName, 'Delete' => ['Objects' => $listOfObjects]], $args);
try {
$this->client->deleteObjects($parameters);
if ($this->verbose) {
echo "Deleted the list of objects from: $bucketName.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to delete the list of objects from $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with object deletion before continuing.";
}
throw $exception;
}
}
public function deleteBucket(string $bucketName, array $args = [])
{
$parameters = array_merge(['Bucket' => $bucketName], $args);
try {
$this->client->deleteBucket($parameters);
if ($this->verbose) {
echo "Deleted the bucket named: $bucketName.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to delete $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with bucket deletion before continuing.";
}
throw $exception;
}
}
public function deleteObject(string $bucketName, string $fileName, array $args = [])
{
$parameters = array_merge(['Bucket' => $bucketName, 'Key' => $fileName], $args);
try {
$this->client->deleteObject($parameters);
if ($this->verbose) {
echo "Deleted the object named: $fileName from $bucketName.\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to delete $fileName from $bucketName with error: {$exception->getMessage()}\n";
echo "Please fix error with object deletion before continuing.";
}
throw $exception;
}
}
public function listBuckets(array $args = [])
{
try {
$buckets = $this->client->listBuckets($args);
if ($this->verbose) {
echo "Retrieved all " . count($buckets) . "\n";
}
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to retrieve bucket list with error: {$exception->getMessage()}\n";
echo "Please fix error with bucket lists before continuing.";
}
throw $exception;
}
return $buckets;
}
public function preSignedUrl(CommandInterface $command, DateTimeInterface|int|string $expires, array $options = [])
{
$request = $this->client->createPresignedRequest($command, $expires, $options);
try {
$presignedUrl = (string)$request->getUri();
} catch (AwsException $exception) {
if ($this->verbose) {
echo "Failed to create a presigned url: {$exception->getMessage()}\n";
echo "Please fix error with presigned urls before continuing.";
}
throw $exception;
}
return $presignedUrl;
}
public function createSession(string $bucketName)
{
try{
$result = $this->client->createSession([
'Bucket' => $bucketName,
]);
return $result;
}catch(S3Exception $caught){
if($caught->getAwsErrorType() == "NoSuchBucket"){
echo "The specified bucket does not exist.";
}
throw $caught;
}
}
}
```
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/s3/s3_basics#code-examples).
Generate a presigned URL that can perform an S3 action for a limited time. Use the Requests package to make a request with the URL.
```
import argparse
import logging
import boto3
from botocore.exceptions import ClientError
import requests
logger = logging.getLogger(__name__)
def generate_presigned_url(s3_client, client_method, method_parameters, expires_in):
"""
Generate a presigned Amazon S3 URL that can be used to perform an action.
:param s3_client: A Boto3 Amazon S3 client.
:param client_method: The name of the client method that the URL performs.
:param method_parameters: The parameters of the specified client method.
:param expires_in: The number of seconds the presigned URL is valid for.
:return: The presigned URL.
"""
try:
url = s3_client.generate_presigned_url(
ClientMethod=client_method, Params=method_parameters, ExpiresIn=expires_in
)
logger.info("Got presigned URL: %s", url)
except ClientError:
logger.exception(
"Couldn't get a presigned URL for client method '%s'.", client_method
)
raise
return url
def usage_demo():
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
print("-" * 88)
print("Welcome to the Amazon S3 presigned URL demo.")
print("-" * 88)
parser = argparse.ArgumentParser()
parser.add_argument("bucket", help="The name of the bucket.")
parser.add_argument(
"key",
help="For a GET operation, the key of the object in Amazon S3. For a "
"PUT operation, the name of a file to upload.",
)
parser.add_argument("action", choices=("get", "put"), help="The action to perform.")
args = parser.parse_args()
s3_client = boto3.client("s3")
client_action = "get_object" if args.action == "get" else "put_object"
url = generate_presigned_url(
s3_client, client_action, {"Bucket": args.bucket, "Key": args.key}, 1000
)
print("Using the Requests package to send a request to the URL.")
response = None
if args.action == "get":
response = requests.get(url)
if response.status_code == 200:
with open(args.key.split("/")[-1], 'wb') as object_file:
object_file.write(response.content)
elif args.action == "put":
print("Putting data to the URL.")
try:
with open(args.key, "rb") as object_file:
object_text = object_file.read()
response = requests.put(url, data=object_text)
except FileNotFoundError:
print(
f"Couldn't find {args.key}. For a PUT operation, the key must be the "
f"name of a file that exists on your computer."
)
if response is not None:
print(f"Status: {response.status_code}\nReason: {response.reason}")
print("-" * 88)
if __name__ == "__main__":
usage_demo()
```
Generate a presigned POST request to upload a file.
```
class BucketWrapper:
"""Encapsulates S3 bucket actions."""
def __init__(self, bucket):
"""
:param bucket: A Boto3 Bucket resource. This is a high-level resource in Boto3
that wraps bucket actions in a class-like structure.
"""
self.bucket = bucket
self.name = bucket.name
def generate_presigned_post(self, object_key, expires_in):
"""
Generate a presigned Amazon S3 POST request to upload a file.
A presigned POST can be used for a limited time to let someone without an AWS
account upload a file to a bucket.
:param object_key: The object key to identify the uploaded object.
:param expires_in: The number of seconds the presigned POST is valid.
:return: A dictionary that contains the URL and form fields that contain
required access data.
"""
try:
response = self.bucket.meta.client.generate_presigned_post(
Bucket=self.bucket.name, Key=object_key, ExpiresIn=expires_in
)
logger.info("Got presigned POST URL: %s", response["url"])
except ClientError:
logger.exception(
"Couldn't get a presigned POST URL for bucket '%s' and object '%s'",
self.bucket.name,
object_key,
)
raise
return response
```
------
#### [ Ruby ]
**SDK for Ruby**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/ruby/example_code/s3#code-examples).
```
require 'aws-sdk-s3'
require 'net/http'
# Creates a presigned URL that can be used to upload content to an object.
#
# @param bucket [Aws::S3::Bucket] An existing Amazon S3 bucket.
# @param object_key [String] The key to give the uploaded object.
# @return [URI, nil] The parsed URI if successful; otherwise nil.
def get_presigned_url(bucket, object_key)
url = bucket.object(object_key).presigned_url(:put)
puts "Created presigned URL: #{url}"
URI(url)
rescue Aws::Errors::ServiceError => e
puts "Couldn't create presigned URL for #{bucket.name}:#{object_key}. Here's why: #{e.message}"
end
# Example usage:
def run_demo
bucket_name = "amzn-s3-demo-bucket"
object_key = "my-file.txt"
object_content = "This is the content of my-file.txt."
bucket = Aws::S3::Bucket.new(bucket_name)
presigned_url = get_presigned_url(bucket, object_key)
return unless presigned_url
response = Net::HTTP.start(presigned_url.host) do |http|
http.send_request('PUT', presigned_url.request_uri, object_content, 'content_type' => '')
end
case response
when Net::HTTPSuccess
puts 'Content uploaded!'
else
puts response.value
end
end
run_demo if $PROGRAM_NAME == __FILE__
```
------
#### [ Rust ]
**SDK for Rust**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/rustv1/examples/s3#code-examples).
Create presigning requests to GET S3 objects.
```
/// Generate a URL for a presigned GET request.
async fn get_object(
client: &Client,
bucket: &str,
object: &str,
expires_in: u64,
) -> Result<(), Box> {
let expires_in = Duration::from_secs(expires_in);
let presigned_request = client
.get_object()
.bucket(bucket)
.key(object)
.presigned(PresigningConfig::expires_in(expires_in)?)
.await?;
println!("Object URI: {}", presigned_request.uri());
let valid_until = chrono::offset::Local::now() + expires_in;
println!("Valid until: {valid_until}");
Ok(())
}
```
Create presigning requests to PUT S3 objects.
```
async fn put_object(
client: &Client,
bucket: &str,
object: &str,
expires_in: u64,
) -> Result {
let expires_in: std::time::Duration = std::time::Duration::from_secs(expires_in);
let expires_in: aws_sdk_s3::presigning::PresigningConfig =
PresigningConfig::expires_in(expires_in).map_err(|err| {
S3ExampleError::new(format!(
"Failed to convert expiration to PresigningConfig: {err:?}"
))
})?;
let presigned_request = client
.put_object()
.bucket(bucket)
.key(object)
.presigned(expires_in)
.await?;
Ok(presigned_request.uri().into())
}
```
------
#### [ SAP ABAP ]
**SDK for SAP ABAP**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/sap-abap/services/s3#code-examples).
Create presigned requests to GET S3 objects.
```
" iv_bucket_name is the bucket name
" iv_key is the object name like "myfile.txt"
DATA(lo_session) = /aws1/cl_rt_session_aws=>create( cv_pfl ).
DATA(lo_s3) = /aws1/cl_s3_factory=>create( lo_session ).
"Upload a nice Hello World file to an S3 bucket."
TRY.
DATA(lv_contents) = cl_abap_codepage=>convert_to( 'Hello, World' ).
lo_s3->putobject(
iv_bucket = iv_bucket_name
iv_key = iv_key
iv_body = lv_contents
iv_contenttype = 'text/plain' ).
MESSAGE 'Object uploaded to S3 bucket.' TYPE 'I'.
CATCH /aws1/cx_s3_nosuchbucket.
MESSAGE 'Bucket does not exist.' TYPE 'E'.
ENDTRY.
" now generate a presigned URL with a 600-second expiration
DATA(lo_presigner) = lo_s3->get_presigner( iv_expires_sec = 600 ).
" the presigner getobject() method has the same signature as
" lo_s3->getobject(), but it doesn't actually make the call.
" to the service. It just prepares a presigned URL for a future call
DATA(lo_presigned_req) = lo_presigner->getobject(
iv_bucket = iv_bucket_name
iv_key = iv_key ).
" You can provide this URL to a web page, user, email etc so they
" can retrieve the file. The URL will expire in 10 minutes.
ov_url = lo_presigned_req->get_url( ).
```
------
# Create a photo asset management application that lets users manage photos using labels
The following code examples show how to create a serverless application that lets users manage photos using labels.
------
#### [ .NET ]
**SDK for .NET**
Shows how to develop a photo asset management application that detects labels in images using Amazon Rekognition and stores them for later retrieval.
For complete source code and instructions on how to set up and run, see the full example on [ GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/cross-service/PhotoAssetManager).
For a deep dive into the origin of this example see the post on [AWS Community](https://community.aws/posts/cloud-journeys/01-serverless-image-recognition-app).
**Services used in this example**
+ API Gateway
+ DynamoDB
+ Lambda
+ Amazon Rekognition
+ Amazon S3
+ Amazon SNS
------
#### [ C\$1\$1 ]
**SDK for C\$1\$1**
Shows how to develop a photo asset management application that detects labels in images using Amazon Rekognition and stores them for later retrieval.
For complete source code and instructions on how to set up and run, see the full example on [ GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/cpp/example_code/cross-service/photo_asset_manager).
For a deep dive into the origin of this example see the post on [AWS Community](https://community.aws/posts/cloud-journeys/01-serverless-image-recognition-app).
**Services used in this example**
+ API Gateway
+ DynamoDB
+ Lambda
+ Amazon Rekognition
+ Amazon S3
+ Amazon SNS
------
#### [ Java ]
**SDK for Java 2.x**
Shows how to develop a photo asset management application that detects labels in images using Amazon Rekognition and stores them for later retrieval.
For complete source code and instructions on how to set up and run, see the full example on [ GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/usecases/pam_source_files).
For a deep dive into the origin of this example see the post on [AWS Community](https://community.aws/posts/cloud-journeys/01-serverless-image-recognition-app).
**Services used in this example**
+ API Gateway
+ DynamoDB
+ Lambda
+ Amazon Rekognition
+ Amazon S3
+ Amazon SNS
------
#### [ JavaScript ]
**SDK for JavaScript (v3)**
Shows how to develop a photo asset management application that detects labels in images using Amazon Rekognition and stores them for later retrieval.
For complete source code and instructions on how to set up and run, see the full example on [ GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javascriptv3/example_code/cross-services/photo-asset-manager).
For a deep dive into the origin of this example see the post on [AWS Community](https://community.aws/posts/cloud-journeys/01-serverless-image-recognition-app).
**Services used in this example**
+ API Gateway
+ DynamoDB
+ Lambda
+ Amazon Rekognition
+ Amazon S3
+ Amazon SNS
------
#### [ Kotlin ]
**SDK for Kotlin**
Shows how to develop a photo asset management application that detects labels in images using Amazon Rekognition and stores them for later retrieval.
For complete source code and instructions on how to set up and run, see the full example on [ GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/kotlin/usecases/creating_pam).
For a deep dive into the origin of this example see the post on [AWS Community](https://community.aws/posts/cloud-journeys/01-serverless-image-recognition-app).
**Services used in this example**
+ API Gateway
+ DynamoDB
+ Lambda
+ Amazon Rekognition
+ Amazon S3
+ Amazon SNS
------
#### [ PHP ]
**SDK for PHP**
Shows how to develop a photo asset management application that detects labels in images using Amazon Rekognition and stores them for later retrieval.
For complete source code and instructions on how to set up and run, see the full example on [ GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/php/applications/photo_asset_manager).
For a deep dive into the origin of this example see the post on [AWS Community](https://community.aws/posts/cloud-journeys/01-serverless-image-recognition-app).
**Services used in this example**
+ API Gateway
+ DynamoDB
+ Lambda
+ Amazon Rekognition
+ Amazon S3
+ Amazon SNS
------
#### [ Rust ]
**SDK for Rust**
Shows how to develop a photo asset management application that detects labels in images using Amazon Rekognition and stores them for later retrieval.
For complete source code and instructions on how to set up and run, see the full example on [ GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/rustv1/cross_service/photo_asset_management).
For a deep dive into the origin of this example see the post on [AWS Community](https://community.aws/posts/cloud-journeys/01-serverless-image-recognition-app).
**Services used in this example**
+ API Gateway
+ DynamoDB
+ Lambda
+ Amazon Rekognition
+ Amazon S3
+ Amazon SNS
------
# A web page that lists Amazon S3 objects using an AWS SDK
The following code example shows how to list Amazon S3 objects in a web page.
------
#### [ JavaScript ]
**SDK for JavaScript (v3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javascriptv3/example_code/web/s3/list-objects#code-examples).
The following code is the relevant React component that makes calls to the AWS SDK. A runnable version of the application containing this component can be found at the preceding GitHub link.
```
import { useEffect, useState } from "react";
import {
ListObjectsCommand,
type ListObjectsCommandOutput,
S3Client,
} from "@aws-sdk/client-s3";
import { fromCognitoIdentityPool } from "@aws-sdk/credential-providers";
import "./App.css";
function App() {
const [objects, setObjects] = useState<
Required["Contents"]
>([]);
useEffect(() => {
const client = new S3Client({
region: "us-east-1",
// Unless you have a public bucket, you'll need access to a private bucket.
// One way to do this is to create an Amazon Cognito identity pool, attach a role to the pool,
// and grant the role access to the 's3:GetObject' action.
//
// You'll also need to configure the CORS settings on the bucket to allow traffic from
// this example site. Here's an example configuration that allows all origins. Don't
// do this in production.
//[
// {
// "AllowedHeaders": ["*"],
// "AllowedMethods": ["GET"],
// "AllowedOrigins": ["*"],
// "ExposeHeaders": [],
// },
//]
//
credentials: fromCognitoIdentityPool({
clientConfig: { region: "us-east-1" },
identityPoolId: "",
}),
});
const command = new ListObjectsCommand({ Bucket: "bucket-name" });
client.send(command).then(({ Contents }) => setObjects(Contents || []));
}, []);
return (
{objects.map((o) => (
{o.Key}
))}
* This method retrieves a list of all incomplete multipart uploads in the specified S3 bucket,
* and then aborts each of those uploads.
*/
public static void abortIncompleteMultipartUploadsFromList() {
ListMultipartUploadsRequest listMultipartUploadsRequest = ListMultipartUploadsRequest.builder()
.bucket(bucketName)
.build();
ListMultipartUploadsResponse response = s3Client.listMultipartUploads(listMultipartUploadsRequest);
List uploads = response.uploads();
AbortMultipartUploadRequest abortMultipartUploadRequest;
for (MultipartUpload upload : uploads) {
abortMultipartUploadRequest = AbortMultipartUploadRequest.builder()
.bucket(bucketName)
.key(upload.key())
.expectedBucketOwner(accountId)
.uploadId(upload.uploadId())
.build();
AbortMultipartUploadResponse abortMultipartUploadResponse = s3Client.abortMultipartUpload(abortMultipartUploadRequest);
if (abortMultipartUploadResponse.sdkHttpResponse().isSuccessful()) {
logger.info("Upload ID [{}] to bucket [{}] successfully aborted.", upload.uploadId(), bucketName);
}
}
}
```
To delete incomplete multipart uploads that were initiated before or after a date, you can selectively delete multipart uploads based on a point in time as shown in the following example.
```
static void abortIncompleteMultipartUploadsOlderThan(Instant pointInTime) {
ListMultipartUploadsRequest listMultipartUploadsRequest = ListMultipartUploadsRequest.builder()
.bucket(bucketName)
.build();
ListMultipartUploadsResponse response = s3Client.listMultipartUploads(listMultipartUploadsRequest);
List uploads = response.uploads();
AbortMultipartUploadRequest abortMultipartUploadRequest;
for (MultipartUpload upload : uploads) {
logger.info("Found multipartUpload with upload ID [{}], initiated [{}]", upload.uploadId(), upload.initiated());
if (upload.initiated().isBefore(pointInTime)) {
abortMultipartUploadRequest = AbortMultipartUploadRequest.builder()
.bucket(bucketName)
.key(upload.key())
.expectedBucketOwner(accountId)
.uploadId(upload.uploadId())
.build();
AbortMultipartUploadResponse abortMultipartUploadResponse = s3Client.abortMultipartUpload(abortMultipartUploadRequest);
if (abortMultipartUploadResponse.sdkHttpResponse().isSuccessful()) {
logger.info("Upload ID [{}] to bucket [{}] successfully aborted.", upload.uploadId(), bucketName);
}
}
}
}
```
If you have access to the upload ID after you begin a multipart upload, you can delete the in-progress upload by using the ID.
```
static void abortMultipartUploadUsingUploadId() {
String uploadId = startUploadReturningUploadId();
AbortMultipartUploadResponse response = s3Client.abortMultipartUpload(b -> b
.uploadId(uploadId)
.bucket(bucketName)
.key(key));
if (response.sdkHttpResponse().isSuccessful()) {
logger.info("Upload ID [{}] to bucket [{}] successfully aborted.", uploadId, bucketName);
}
}
```
To consistently delete incomplete multipart uploads older that a certain number of days, set up a bucket lifecycle configuration for the bucket. The following example shows how to create a rule to delete incomplete uploads older than 7 days.
```
static void abortMultipartUploadsUsingLifecycleConfig() {
Collection lifeCycleRules = List.of(LifecycleRule.builder()
.abortIncompleteMultipartUpload(b -> b.
daysAfterInitiation(7))
.status("Enabled")
.filter(SdkBuilder::build) // Filter element is required.
.build());
// If the action is successful, the service sends back an HTTP 200 response with an empty HTTP body.
PutBucketLifecycleConfigurationResponse response = s3Client.putBucketLifecycleConfiguration(b -> b
.bucket(bucketName)
.lifecycleConfiguration(b1 -> b1.rules(lifeCycleRules)));
if (response.sdkHttpResponse().isSuccessful()) {
logger.info("Rule to abort incomplete multipart uploads added to bucket.");
} else {
logger.error("Unsuccessfully applied rule. HTTP status code is [{}]", response.sdkHttpResponse().statusCode());
}
}
```
+ For API details, see the following topics in *AWS SDK for Java 2.x API Reference*.
+ [AbortMultipartUpload](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/AbortMultipartUpload)
+ [ListMultipartUploads](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/ListMultipartUploads)
+ [PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutBucketLifecycleConfiguration)
------
# Detect PPE in images with Amazon Rekognition using an AWS SDK
The following code example shows how to build an app that uses Amazon Rekognition to detect Personal Protective Equipment (PPE) in images.
------
#### [ Java ]
**SDK for Java 2.x**
Shows how to create an AWS Lambda function that detects images with Personal Protective Equipment.
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/usecases/creating_lambda_ppe).
**Services used in this example**
+ DynamoDB
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
------
# Detect entities in text extracted from an image using an AWS SDK
The following code example shows how to use Amazon Comprehend to detect entities in text extracted by Amazon Textract from an image that is stored in Amazon S3.
------
#### [ Python ]
**SDK for Python (Boto3)**
Shows how to use the AWS SDK for Python (Boto3) in a Jupyter notebook to detect entities in text that is extracted from an image. This example uses Amazon Textract to extract text from an image stored in Amazon Simple Storage Service (Amazon S3) and Amazon Comprehend to detect entities in the extracted text.
This example is a Jupyter notebook and must be run in an environment that can host notebooks. For instructions on how to run the example using Amazon SageMaker AI, see the directions in [TextractAndComprehendNotebook.ipynb](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/cross_service/textract_comprehend_notebook/TextractAndComprehendNotebook.ipynb).
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/cross_service/textract_comprehend_notebook#readme).
**Services used in this example**
+ Amazon Comprehend
+ Amazon S3
+ Amazon Textract
------
# Detect faces in an image using an AWS SDK
The following code example shows how to:
+ Save an image in an Amazon S3 bucket.
+ Use Amazon Rekognition to detect facial details, such as age range, gender, and emotion (such as smiling).
+ Display those details.
------
#### [ Rust ]
**SDK for Rust**
Save the image in an Amazon S3 bucket with an **uploads** prefix, use Amazon Rekognition to detect facial details, such as age range, gender, and emotion (smiling, etc.), and display those details.
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/rustv1/cross_service/detect_faces/src/main.rs).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
------
# Detect objects in images with Amazon Rekognition using an AWS SDK
The following code examples show how to build an app that uses Amazon Rekognition to detect objects by category in images.
------
#### [ .NET ]
**SDK for .NET**
Shows how to use Amazon Rekognition .NET API to create an app that uses Amazon Rekognition to identify objects by category in images located in an Amazon Simple Storage Service (Amazon S3) bucket. The app sends the admin an email notification with the results using Amazon Simple Email Service (Amazon SES).
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/cross-service/PhotoAnalyzerApp).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
------
#### [ Java ]
**SDK for Java 2.x**
Shows how to use Amazon Rekognition Java API to create an app that uses Amazon Rekognition to identify objects by category in images located in an Amazon Simple Storage Service (Amazon S3) bucket. The app sends the admin an email notification with the results using Amazon Simple Email Service (Amazon SES).
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/usecases/creating_photo_analyzer_app).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
------
#### [ JavaScript ]
**SDK for JavaScript (v3)**
Shows how to use Amazon Rekognition with the AWS SDK for JavaScript to create an app that uses Amazon Rekognition to identify objects by category in images located in an Amazon Simple Storage Service (Amazon S3) bucket. The app sends the admin an email notification with the results using Amazon Simple Email Service (Amazon SES).
Learn how to:
+ Create an unauthenticated user using Amazon Cognito.
+ Analyze images for objects using Amazon Rekognition.
+ Verify an email address for Amazon SES.
+ Send an email notification using Amazon SES.
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javascriptv3/example_code/cross-services/photo_analyzer).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
------
#### [ Kotlin ]
**SDK for Kotlin**
Shows how to use Amazon Rekognition Kotlin API to create an app that uses Amazon Rekognition to identify objects by category in images located in an Amazon Simple Storage Service (Amazon S3) bucket. The app sends the admin an email notification with the results using Amazon Simple Email Service (Amazon SES).
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/kotlin/usecases/creating_photo_analyzer_app).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
------
#### [ Python ]
**SDK for Python (Boto3)**
Shows you how to use the AWS SDK for Python (Boto3) to create a web application that lets you do the following:
+ Upload photos to an Amazon Simple Storage Service (Amazon S3) bucket.
+ Use Amazon Rekognition to analyze and label the photos.
+ Use Amazon Simple Email Service (Amazon SES) to send email reports of image analysis.
This example contains two main components: a webpage written in JavaScript that is built with React, and a REST service written in Python that is built with Flask-RESTful.
You can use the React webpage to:
+ Display a list of images that are stored in your S3 bucket.
+ Upload images from your computer to your S3 bucket.
+ Display images and labels that identify items that are detected in the image.
+ Get a report of all images in your S3 bucket and send an email of the report.
The webpage calls the REST service. The service sends requests to AWS to perform the following actions:
+ Get and filter the list of images in your S3 bucket.
+ Upload photos to your S3 bucket.
+ Use Amazon Rekognition to analyze individual photos and get a list of labels that identify items that are detected in the photo.
+ Analyze all photos in your S3 bucket and use Amazon SES to email a report.
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/cross_service/photo_analyzer).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
------
# Detect people and objects in a video with Amazon Rekognition using an AWS SDK
The following code examples show how to detect people and objects in a video with Amazon Rekognition.
------
#### [ Java ]
**SDK for Java 2.x**
Shows how to use Amazon Rekognition Java API to create an app to detect faces and objects in videos located in an Amazon Simple Storage Service (Amazon S3) bucket. The app sends the admin an email notification with the results using Amazon Simple Email Service (Amazon SES).
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/usecases/video_analyzer_application).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
+ Amazon SNS
+ Amazon SQS
------
#### [ Python ]
**SDK for Python (Boto3)**
Use Amazon Rekognition to detect faces, objects, and people in videos by starting asynchronous detection jobs. This example also configures Amazon Rekognition to notify an Amazon Simple Notification Service (Amazon SNS) topic when jobs complete and subscribes an Amazon Simple Queue Service (Amazon SQS) queue to the topic. When the queue receives a message about a job, the job is retrieved and the results are output.
This example is best viewed on GitHub. For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/rekognition).
**Services used in this example**
+ Amazon Rekognition
+ Amazon S3
+ Amazon SES
+ Amazon SNS
+ Amazon SQS
------
# Download S3 'directories' from an Amazon Simple Storage Service (Amazon S3) bucket
The following code example shows how to download and filter the contents of Amazon S3 bucket 'directories'.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
This example show how to use the [S3TransferManager](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/S3TransferManager.html) in the AWS SDK for Java 2.x to download 'directories' from an Amazon S3 bucket. It also demonstrates how to use [DownloadFilters](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/config/DownloadFilter.html) in the request.
```
/**
* For standard buckets, S3 provides the illusion of a directory structure through the use of keys. When you upload
* an object to an S3 bucket, you specify a key, which is essentially the "path" to the object. The key can contain
* forward slashes ("/") to make it appear as if the object is stored in a directory structure, but this is just a
* logical representation, not an actual directory.
*
* In this example, our S3 bucket contains the following objects:
*
* folder1/file1.txt
* folder1/file2.txt
* folder1/file3.txt
* folder2/file1.txt
* folder2/file2.txt
* folder2/file3.txt
* folder3/file1.txt
* folder3/file2.txt
* folder3/file3.txt
*
* When method `downloadS3Directories` is invoked with
* `destinationPathURI` set to `/test`, the downloaded
* directory looks like:
*
* |- test
* |- folder1
* |- file1.txt
* |- file2.txt
* |- file3.txt
* |- folder3
* |- file1.txt
* |- file2.txt
* |- file3.txt
*
*
* @param transferManager An S3TransferManager instance.
* @param destinationPathURI local directory to hold the downloaded S3 'directories' and files.
* @param bucketName The S3 bucket that contains the 'directories' to download.
* @return The number of objects (files, in this case) that were downloaded.
*/
public Integer downloadS3Directories(S3TransferManager transferManager,
URI destinationPathURI, String bucketName) {
// Define the filters for which 'directories' we want to download.
DownloadFilter folder1Filter = (S3Object s3Object) -> s3Object.key().startsWith("folder1/");
DownloadFilter folder3Filter = (S3Object s3Object) -> s3Object.key().startsWith("folder3/");
DownloadFilter folderFilter = s3Object -> folder1Filter.or(folder3Filter).test(s3Object);
DirectoryDownload directoryDownload = transferManager.downloadDirectory(DownloadDirectoryRequest.builder()
.destination(Paths.get(destinationPathURI))
.bucket(bucketName)
.filter(folderFilter)
.build());
CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();
Integer numFilesInFolder1 = Paths.get(destinationPathURI).resolve("folder1").toFile().list().length;
Integer numFilesInFolder3 = Paths.get(destinationPathURI).resolve("folder3").toFile().list().length;
try {
assert numFilesInFolder1 == 3;
assert numFilesInFolder3 == 3;
assert !Paths.get(destinationPathURI).resolve("folder2").toFile().exists(); // `folder2` was not downloaded.
} catch (AssertionError e) {
logger.error("An assertion failed.");
}
completedDirectoryDownload.failedTransfers()
.forEach(fail -> logger.warn("Object failed to transfer [{}]", fail.exception().getMessage()));
return numFilesInFolder1 + numFilesInFolder3;
}
```
------
# Download all objects in an Amazon Simple Storage Service (Amazon S3) bucket to a local directory
The following code example shows how to download all objects in an Amazon Simple Storage Service (Amazon S3) bucket to a local directory.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
Use an [S3TransferManager](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/S3TransferManager.html) to [download all S3 objects](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/S3TransferManager.html#downloadDirectory(software.amazon.awssdk.transfer.s3.DownloadDirectoryRequest)) in the same S3 bucket. View the [complete file](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/s3/src/main/java/com/example/s3/transfermanager/DownloadToDirectory.java) and [test](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/s3/src/test/java/TransferManagerTest.java).
```
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.model.ObjectIdentifier;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.CompletedDirectoryDownload;
import software.amazon.awssdk.transfer.s3.model.DirectoryDownload;
import software.amazon.awssdk.transfer.s3.model.DownloadDirectoryRequest;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashSet;
import java.util.Set;
import java.util.UUID;
import java.util.stream.Collectors;
public Integer downloadObjectsToDirectory(S3TransferManager transferManager,
URI destinationPathURI, String bucketName) {
DirectoryDownload directoryDownload = transferManager.downloadDirectory(DownloadDirectoryRequest.builder()
.destination(Paths.get(destinationPathURI))
.bucket(bucketName)
.build());
CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();
completedDirectoryDownload.failedTransfers()
.forEach(fail -> logger.warn("Object [{}] failed to transfer", fail.toString()));
return completedDirectoryDownload.failedTransfers().size();
}
```
+ For API details, see [DownloadDirectory](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/DownloadDirectory) in *AWS SDK for Java 2.x API Reference*.
------
# Download a stream of unknown size from an Amazon S3 object using an AWS SDK
The following code example shows how to download a stream of unknown size from an Amazon S3 object.
------
#### [ Swift ]
**SDK for Swift**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/swift/example_code/s3/binary-streaming#code-examples).
```
import ArgumentParser
import AWSClientRuntime
import AWSS3
import Foundation
import Smithy
import SmithyHTTPAPI
import SmithyStreams
/// Download a file from the specified bucket.
///
/// - Parameters:
/// - bucket: The Amazon S3 bucket name to get the file from.
/// - key: The name (or path) of the file to download from the bucket.
/// - destPath: The pathname on the local filesystem at which to store
/// the downloaded file.
func downloadFile(bucket: String, key: String, destPath: String?) async throws {
let fileURL: URL
// If no destination path was provided, use the key as the name to use
// for the file in the downloads folder.
if destPath == nil {
do {
try fileURL = FileManager.default.url(
for: .downloadsDirectory,
in: .userDomainMask,
appropriateFor: URL(string: key),
create: true
).appendingPathComponent(key)
} catch {
throw TransferError.directoryError
}
} else {
fileURL = URL(fileURLWithPath: destPath!)
}
let config = try await S3Client.S3ClientConfiguration(region: region)
let s3Client = S3Client(config: config)
// Create a `FileHandle` referencing the local destination. Then
// create a `ByteStream` from that.
FileManager.default.createFile(atPath: fileURL.path, contents: nil, attributes: nil)
let fileHandle = try FileHandle(forWritingTo: fileURL)
// Download the file using `GetObject`.
let getInput = GetObjectInput(
bucket: bucket,
key: key
)
do {
let getOutput = try await s3Client.getObject(input: getInput)
guard let body = getOutput.body else {
throw TransferError.downloadError("Error: No data returned for download")
}
// If the body is returned as a `Data` object, write that to the
// file. If it's a stream, read the stream chunk by chunk,
// appending each chunk to the destination file.
switch body {
case .data:
guard let data = try await body.readData() else {
throw TransferError.downloadError("Download error")
}
// Write the `Data` to the file.
do {
try data.write(to: fileURL)
} catch {
throw TransferError.writeError
}
break
case .stream(let stream as ReadableStream):
while (true) {
let chunk = try await stream.readAsync(upToCount: 5 * 1024 * 1024)
guard let chunk = chunk else {
break
}
// Write the chunk to the destination file.
do {
try fileHandle.write(contentsOf: chunk)
} catch {
throw TransferError.writeError
}
}
break
default:
throw TransferError.downloadError("Received data is unknown object type")
}
} catch {
throw TransferError.downloadError("Error downloading the file: \(error)")
}
print("File downloaded to \(fileURL.path).")
}
```
------
# Get an Amazon S3 object from a Multi-Region Access Point by using an AWS SDK
The following code example shows how to get an object from a Multi-Region Access Point.
------
#### [ Kotlin ]
**SDK for Kotlin**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/kotlin/services/s3#code-examples).
Configure the S3 client to use the Asymmetric Sigv4 (Sigv4a) signing algorithm.
```
suspend fun createS3Client(): S3Client {
// Configure your S3Client to use the Asymmetric SigV4 (SigV4a) signing algorithm.
val sigV4aScheme = SigV4AsymmetricAuthScheme(DefaultAwsSigner)
val s3 = S3Client.fromEnvironment {
authSchemes = listOf(sigV4aScheme)
}
return s3
}
```
Use the Multi-Region Access Point ARN instead of a bucket name to retrieve the object.
```
suspend fun getObjectFromMrap(
s3: S3Client,
mrapArn: String,
keyName: String,
): String? {
val request = GetObjectRequest {
bucket = mrapArn // Use the ARN instead of the bucket name for object operations.
key = keyName
}
var stringObj: String? = null
s3.getObject(request) { resp ->
stringObj = resp.body?.decodeToString()
if (stringObj != null) {
println("Successfully read $keyName from $mrapArn")
}
}
return stringObj
}
```
+ For more information, see [AWS SDK for Kotlin developer guide](https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/use-services-s3-mrap.html).
+ For API details, see [GetObject](https://sdk.amazonaws.com/kotlin/api/latest/index.html) in *AWS SDK for Kotlin API reference*.
------
# Get an object from an Amazon S3 bucket using an AWS SDK, specifying an If-Modified-Since header
The following code example shows how to read data from an object in an S3 bucket, but only if that bucket has not been modified since the last retrieval time.
------
#### [ Rust ]
**SDK for Rust**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/rustv1/examples/s3#code-examples).
```
use aws_sdk_s3::{
error::SdkError,
primitives::{ByteStream, DateTime, DateTimeFormat},
Client,
};
use s3_code_examples::error::S3ExampleError;
use tracing::{error, warn};
const KEY: &str = "key";
const BODY: &str = "Hello, world!";
/// Demonstrate how `if-modified-since` reports that matching objects haven't
/// changed.
///
/// # Steps
/// - Create a bucket.
/// - Put an object in the bucket.
/// - Get the bucket headers.
/// - Get the bucket headers again but only if modified.
/// - Delete the bucket.
#[tokio::main]
async fn main() -> Result<(), S3ExampleError> {
tracing_subscriber::fmt::init();
// Get a new UUID to use when creating a unique bucket name.
let uuid = uuid::Uuid::new_v4();
// Load the AWS configuration from the environment.
let client = Client::new(&aws_config::load_from_env().await);
// Generate a unique bucket name using the previously generated UUID.
// Then create a new bucket with that name.
let bucket_name = format!("if-modified-since-{uuid}");
client
.create_bucket()
.bucket(bucket_name.clone())
.send()
.await?;
// Create a new object in the bucket whose name is `KEY` and whose
// contents are `BODY`.
let put_object_output = client
.put_object()
.bucket(bucket_name.as_str())
.key(KEY)
.body(ByteStream::from_static(BODY.as_bytes()))
.send()
.await;
// If the `PutObject` succeeded, get the eTag string from it. Otherwise,
// report an error and return an empty string.
let e_tag_1 = match put_object_output {
Ok(put_object) => put_object.e_tag.unwrap(),
Err(err) => {
error!("{err:?}");
String::new()
}
};
// Request the object's headers.
let head_object_output = client
.head_object()
.bucket(bucket_name.as_str())
.key(KEY)
.send()
.await;
// If the `HeadObject` request succeeded, create a tuple containing the
// values of the headers `last-modified` and `etag`. If the request
// failed, return the error in a tuple instead.
let (last_modified, e_tag_2) = match head_object_output {
Ok(head_object) => (
Ok(head_object.last_modified().cloned().unwrap()),
head_object.e_tag.unwrap(),
),
Err(err) => (Err(err), String::new()),
};
warn!("last modified: {last_modified:?}");
assert_eq!(
e_tag_1, e_tag_2,
"PutObject and first GetObject had differing eTags"
);
println!("First value of last_modified: {last_modified:?}");
println!("First tag: {}\n", e_tag_1);
// Send a second `HeadObject` request. This time, the `if_modified_since`
// option is specified, giving the `last_modified` value returned by the
// first call to `HeadObject`.
//
// Since the object hasn't been changed, and there are no other objects in
// the bucket, there should be no matching objects.
let head_object_output = client
.head_object()
.bucket(bucket_name.as_str())
.key(KEY)
.if_modified_since(last_modified.unwrap())
.send()
.await;
// If the `HeadObject` request succeeded, the result is a typle containing
// the `last_modified` and `e_tag_1` properties. This is _not_ the expected
// result.
//
// The _expected_ result of the second call to `HeadObject` is an
// `SdkError::ServiceError` containing the HTTP error response. If that's
// the case and the HTTP status is 304 (not modified), the output is a
// tuple containing the values of the HTTP `last-modified` and `etag`
// headers.
//
// If any other HTTP error occurred, the error is returned as an
// `SdkError::ServiceError`.
let (last_modified, e_tag_2) = match head_object_output {
Ok(head_object) => (
Ok(head_object.last_modified().cloned().unwrap()),
head_object.e_tag.unwrap(),
),
Err(err) => match err {
SdkError::ServiceError(err) => {
// Get the raw HTTP response. If its status is 304, the
// object has not changed. This is the expected code path.
let http = err.raw();
match http.status().as_u16() {
// If the HTTP status is 304: Not Modified, return a
// tuple containing the values of the HTTP
// `last-modified` and `etag` headers.
304 => (
Ok(DateTime::from_str(
http.headers().get("last-modified").unwrap(),
DateTimeFormat::HttpDate,
)
.unwrap()),
http.headers().get("etag").map(|t| t.into()).unwrap(),
),
// Any other HTTP status code is returned as an
// `SdkError::ServiceError`.
_ => (Err(SdkError::ServiceError(err)), String::new()),
}
}
// Any other kind of error is returned in a tuple containing the
// error and an empty string.
_ => (Err(err), String::new()),
},
};
warn!("last modified: {last_modified:?}");
assert_eq!(
e_tag_1, e_tag_2,
"PutObject and second HeadObject had different eTags"
);
println!("Second value of last modified: {last_modified:?}");
println!("Second tag: {}", e_tag_2);
// Clean up by deleting the object and the bucket.
client
.delete_object()
.bucket(bucket_name.as_str())
.key(KEY)
.send()
.await?;
client
.delete_bucket()
.bucket(bucket_name.as_str())
.send()
.await?;
Ok(())
}
```
+ For API details, see [GetObject](https://docs.rs/aws-sdk-s3/latest/aws_sdk_s3/client/struct.Client.html#method.get_object) in *AWS SDK for Rust API reference*.
------
# Get started with encryption for Amazon S3 objects using an AWS SDK
The following code example shows how to get started with encryption for Amazon S3 objects.
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/SSEClientEncryptionExample#code-examples).
```
using System;
using System.IO;
using System.Security.Cryptography;
using System.Threading.Tasks;
using Amazon.S3;
using Amazon.S3.Model;
///
/// This example shows how to apply client encryption to an object in an
/// Amazon Simple Storage Service (Amazon S3) bucket.
///
public class SSEClientEncryption
{
public static async Task Main()
{
string bucketName = "amzn-s3-demo-bucket";
string keyName = "exampleobject.txt";
string copyTargetKeyName = "examplecopy.txt";
// If the AWS Region defined for your default user is different
// from the Region where your Amazon S3 bucket is located,
// pass the Region name to the Amazon S3 client object's constructor.
// For example: RegionEndpoint.USWest2.
IAmazonS3 client = new AmazonS3Client();
try
{
// Create an encryption key.
Aes aesEncryption = Aes.Create();
aesEncryption.KeySize = 256;
aesEncryption.GenerateKey();
string base64Key = Convert.ToBase64String(aesEncryption.Key);
// Upload the object.
PutObjectRequest putObjectRequest = await UploadObjectAsync(client, bucketName, keyName, base64Key);
// Download the object and verify that its contents match what you uploaded.
await DownloadObjectAsync(client, bucketName, keyName, base64Key, putObjectRequest);
// Get object metadata and verify that the object uses AES-256 encryption.
await GetObjectMetadataAsync(client, bucketName, keyName, base64Key);
// Copy both the source and target objects using server-side encryption with
// an encryption key.
await CopyObjectAsync(client, bucketName, keyName, copyTargetKeyName, aesEncryption, base64Key);
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"Error: {ex.Message}");
}
}
///
/// Uploads an object to an Amazon S3 bucket.
///
/// The initialized Amazon S3 client object used to call
/// PutObjectAsync.
/// The name of the Amazon S3 bucket to which the
/// object will be uploaded.
/// The name of the object to upload to the Amazon S3
/// bucket.
/// The encryption key.
/// The PutObjectRequest object for use by DownloadObjectAsync.
public static async Task UploadObjectAsync(
IAmazonS3 client,
string bucketName,
string keyName,
string base64Key)
{
PutObjectRequest putObjectRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = keyName,
ContentBody = "sample text",
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = base64Key,
};
PutObjectResponse putObjectResponse = await client.PutObjectAsync(putObjectRequest);
return putObjectRequest;
}
///
/// Downloads an encrypted object from an Amazon S3 bucket.
///
/// The initialized Amazon S3 client object used to call
/// GetObjectAsync.
/// The name of the Amazon S3 bucket where the object
/// is located.
/// The name of the Amazon S3 object to download.
/// The encryption key used to encrypt the
/// object.
/// The PutObjectRequest used to upload
/// the object.
public static async Task DownloadObjectAsync(
IAmazonS3 client,
string bucketName,
string keyName,
string base64Key,
PutObjectRequest putObjectRequest)
{
GetObjectRequest getObjectRequest = new GetObjectRequest
{
BucketName = bucketName,
Key = keyName,
// Provide encryption information for the object stored in Amazon S3.
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = base64Key,
};
using (GetObjectResponse getResponse = await client.GetObjectAsync(getObjectRequest))
using (StreamReader reader = new StreamReader(getResponse.ResponseStream))
{
string content = reader.ReadToEnd();
if (string.Compare(putObjectRequest.ContentBody, content) == 0)
{
Console.WriteLine("Object content is same as we uploaded");
}
else
{
Console.WriteLine("Error...Object content is not same.");
}
if (getResponse.ServerSideEncryptionCustomerMethod == ServerSideEncryptionCustomerMethod.AES256)
{
Console.WriteLine("Object encryption method is AES256, same as we set");
}
else
{
Console.WriteLine("Error...Object encryption method is not the same as AES256 we set");
}
}
}
///
/// Retrieves the metadata associated with an Amazon S3 object.
///
/// The initialized Amazon S3 client object used
/// to call GetObjectMetadataAsync.
/// The name of the Amazon S3 bucket containing the
/// object for which we want to retrieve metadata.
/// The name of the object for which we wish to
/// retrieve the metadata.
/// The encryption key associated with the
/// object.
public static async Task GetObjectMetadataAsync(
IAmazonS3 client,
string bucketName,
string keyName,
string base64Key)
{
GetObjectMetadataRequest getObjectMetadataRequest = new GetObjectMetadataRequest
{
BucketName = bucketName,
Key = keyName,
// The object stored in Amazon S3 is encrypted, so provide the necessary encryption information.
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = base64Key,
};
GetObjectMetadataResponse getObjectMetadataResponse = await client.GetObjectMetadataAsync(getObjectMetadataRequest);
Console.WriteLine("The object metadata show encryption method used is: {0}", getObjectMetadataResponse.ServerSideEncryptionCustomerMethod);
}
///
/// Copies an encrypted object from one Amazon S3 bucket to another.
///
/// The initialized Amazon S3 client object used to call
/// CopyObjectAsync.
/// The Amazon S3 bucket containing the object
/// to copy.
/// The name of the object to copy.
/// The Amazon S3 bucket to which the object
/// will be copied.
/// The encryption type to use.
/// The encryption key to use.
public static async Task CopyObjectAsync(
IAmazonS3 client,
string bucketName,
string keyName,
string copyTargetKeyName,
Aes aesEncryption,
string base64Key)
{
aesEncryption.GenerateKey();
string copyBase64Key = Convert.ToBase64String(aesEncryption.Key);
CopyObjectRequest copyRequest = new CopyObjectRequest
{
SourceBucket = bucketName,
SourceKey = keyName,
DestinationBucket = bucketName,
DestinationKey = copyTargetKeyName,
// Information about the source object's encryption.
CopySourceServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
CopySourceServerSideEncryptionCustomerProvidedKey = base64Key,
// Information about the target object's encryption.
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = copyBase64Key,
};
await client.CopyObjectAsync(copyRequest);
}
}
```
+ For API details, see the following topics in *AWS SDK for .NET API Reference*.
+ [CopyObject](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/CopyObject)
+ [GetObject](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObject)
+ [GetObjectMetadata](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObjectMetadata)
------
# Get started with tags for Amazon S3 objects using an AWS SDK
The following code example shows how to get started with tags for Amazon S3 objects.
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/ObjectTagExample#code-examples).
```
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
///
/// This example shows how to work with tags in Amazon Simple Storage
/// Service (Amazon S3) objects.
///
public class ObjectTag
{
public static async Task Main()
{
string bucketName = "amzn-s3-demo-bucket";
string keyName = "newobject.txt";
string filePath = @"*** file path ***";
// Specify your bucket region (an example region is shown).
RegionEndpoint bucketRegion = RegionEndpoint.USWest2;
var client = new AmazonS3Client(bucketRegion);
await PutObjectsWithTagsAsync(client, bucketName, keyName, filePath);
}
///
/// This method uploads an object with tags. It then shows the tag
/// values, changes the tags, and shows the new tags.
///
/// The Initialized Amazon S3 client object used
/// to call the methods to create and change an objects tags.
/// A string representing the name of the
/// bucket where the object will be stored.
/// A string representing the key name of the
/// object to be tagged.
/// The directory location and file name of the
/// object to be uploaded to the Amazon S3 bucket.
public static async Task PutObjectsWithTagsAsync(IAmazonS3 client, string bucketName, string keyName, string filePath)
{
try
{
// Create an object with tags.
var putRequest = new PutObjectRequest
{
BucketName = bucketName,
Key = keyName,
FilePath = filePath,
TagSet = new List
{
new Tag { Key = "Keyx1", Value = "Value1" },
new Tag { Key = "Keyx2", Value = "Value2" },
},
};
PutObjectResponse response = await client.PutObjectAsync(putRequest);
// Now retrieve the new object's tags.
GetObjectTaggingRequest getTagsRequest = new GetObjectTaggingRequest()
{
BucketName = bucketName,
Key = keyName,
};
GetObjectTaggingResponse objectTags = await client.GetObjectTaggingAsync(getTagsRequest);
// Display the tag values.
objectTags.Tagging
.ForEach(t => Console.WriteLine($"Key: {t.Key}, Value: {t.Value}"));
Tagging newTagSet = new Tagging()
{
TagSet = new List
{
new Tag { Key = "Key3", Value = "Value3" },
new Tag { Key = "Key4", Value = "Value4" },
},
};
PutObjectTaggingRequest putObjTagsRequest = new PutObjectTaggingRequest()
{
BucketName = bucketName,
Key = keyName,
Tagging = newTagSet,
};
PutObjectTaggingResponse response2 = await client.PutObjectTaggingAsync(putObjTagsRequest);
// Retrieve the tags again and show the values.
GetObjectTaggingRequest getTagsRequest2 = new GetObjectTaggingRequest()
{
BucketName = bucketName,
Key = keyName,
};
GetObjectTaggingResponse objectTags2 = await client.GetObjectTaggingAsync(getTagsRequest2);
objectTags2.Tagging
.ForEach(t => Console.WriteLine($"Key: {t.Key}, Value: {t.Value}"));
}
catch (AmazonS3Exception ex)
{
Console.WriteLine(
$"Error: '{ex.Message}'");
}
}
}
```
+ For API details, see [GetObjectTagging](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObjectTagging) in *AWS SDK for .NET API Reference*.
------
# Getting started with Amazon Athena
The following code example shows how to:
+ Create an S3 bucket for query results
+ Create a database
+ Create a table
+ Run a query
+ Create and use named queries
+ Clean up resources
------
#### [ Bash ]
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/061-amazon-athena-gs) repository.
```
#!/bin/bash
# Amazon Athena Getting Started Script
# This script demonstrates how to use Amazon Athena with AWS CLI
# It creates a database, table, runs queries, and manages named queries
# Set up logging
LOG_FILE="athena-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "Starting Amazon Athena Getting Started Tutorial..."
echo "Logging to $LOG_FILE"
# Function to handle errors
handle_error() {
echo "ERROR: $1"
echo "Resources created:"
if [ -n "$NAMED_QUERY_ID" ]; then
echo "- Named Query: $NAMED_QUERY_ID"
fi
if [ -n "$DATABASE_NAME" ]; then
echo "- Database: $DATABASE_NAME"
if [ -n "$TABLE_NAME" ]; then
echo "- Table: $TABLE_NAME in $DATABASE_NAME"
fi
fi
if [ -n "$S3_BUCKET" ]; then
echo "- S3 Bucket: $S3_BUCKET"
fi
echo "Exiting..."
exit 1
}
# Generate a random identifier for S3 bucket
RANDOM_ID=$(openssl rand -hex 6)
S3_BUCKET="athena-${RANDOM_ID}"
DATABASE_NAME="mydatabase"
TABLE_NAME="cloudfront_logs"
# Get the current AWS region
AWS_REGION=$(aws configure get region)
if [ -z "$AWS_REGION" ]; then
AWS_REGION="us-east-1"
echo "No AWS region found in configuration, defaulting to $AWS_REGION"
fi
echo "Using AWS Region: $AWS_REGION"
# Create S3 bucket for Athena query results
echo "Creating S3 bucket for Athena query results: $S3_BUCKET"
CREATE_BUCKET_RESULT=$(aws s3 mb "s3://$S3_BUCKET" 2>&1)
if echo "$CREATE_BUCKET_RESULT" | grep -i "error"; then
handle_error "Failed to create S3 bucket: $CREATE_BUCKET_RESULT"
fi
echo "$CREATE_BUCKET_RESULT"
# Step 1: Create a database
echo "Step 1: Creating Athena database: $DATABASE_NAME"
CREATE_DB_RESULT=$(aws athena start-query-execution \
--query-string "CREATE DATABASE IF NOT EXISTS $DATABASE_NAME" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$CREATE_DB_RESULT" | grep -i "error"; then
handle_error "Failed to create database: $CREATE_DB_RESULT"
fi
QUERY_ID=$(echo "$CREATE_DB_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Database creation query ID: $QUERY_ID"
# Wait for database creation to complete
echo "Waiting for database creation to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Database creation completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Database creation failed with status: $QUERY_STATUS"
fi
echo "Database creation in progress, status: $QUERY_STATUS"
sleep 2
done
# Verify the database was created
echo "Verifying database creation..."
LIST_DB_RESULT=$(aws athena list-databases --catalog-name AwsDataCatalog 2>&1)
if echo "$LIST_DB_RESULT" | grep -i "error"; then
handle_error "Failed to list databases: $LIST_DB_RESULT"
fi
echo "$LIST_DB_RESULT"
# Step 2: Create a table
echo "Step 2: Creating Athena table: $TABLE_NAME"
# Replace the region placeholder in the S3 location
CREATE_TABLE_QUERY="CREATE EXTERNAL TABLE IF NOT EXISTS $DATABASE_NAME.$TABLE_NAME (
\`Date\` DATE,
Time STRING,
Location STRING,
Bytes INT,
RequestIP STRING,
Method STRING,
Host STRING,
Uri STRING,
Status INT,
Referrer STRING,
os STRING,
Browser STRING,
BrowserVersion STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
\"input.regex\" = \"^(?!#)([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+([^ ]+)\\\\s+[^\\\\(]+[\\\\(]([^\\\\;]+).*\\\\%20([^\\\\/]+)[\\\\/](.*)$\"
) LOCATION 's3://athena-examples-us-east-1/cloudfront/plaintext/';"
CREATE_TABLE_RESULT=$(aws athena start-query-execution \
--query-string "$CREATE_TABLE_QUERY" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$CREATE_TABLE_RESULT" | grep -i "error"; then
handle_error "Failed to create table: $CREATE_TABLE_RESULT"
fi
QUERY_ID=$(echo "$CREATE_TABLE_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Table creation query ID: $QUERY_ID"
# Wait for table creation to complete
echo "Waiting for table creation to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Table creation completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Table creation failed with status: $QUERY_STATUS"
fi
echo "Table creation in progress, status: $QUERY_STATUS"
sleep 2
done
# Verify the table was created
echo "Verifying table creation..."
LIST_TABLE_RESULT=$(aws athena list-table-metadata \
--catalog-name AwsDataCatalog \
--database-name "$DATABASE_NAME" 2>&1)
if echo "$LIST_TABLE_RESULT" | grep -i "error"; then
handle_error "Failed to list tables: $LIST_TABLE_RESULT"
fi
echo "$LIST_TABLE_RESULT"
# Step 3: Query data
echo "Step 3: Running a query on the table..."
QUERY="SELECT os, COUNT(*) count
FROM $DATABASE_NAME.$TABLE_NAME
WHERE date BETWEEN date '2014-07-05' AND date '2014-08-05'
GROUP BY os"
QUERY_RESULT=$(aws athena start-query-execution \
--query-string "$QUERY" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$QUERY_RESULT" | grep -i "error"; then
handle_error "Failed to run query: $QUERY_RESULT"
fi
QUERY_ID=$(echo "$QUERY_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Query execution ID: $QUERY_ID"
# Wait for query to complete
echo "Waiting for query to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Query completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Query failed with status: $QUERY_STATUS"
fi
echo "Query in progress, status: $QUERY_STATUS"
sleep 2
done
# Get query results
echo "Getting query results..."
RESULTS=$(aws athena get-query-results --query-execution-id "$QUERY_ID" 2>&1)
if echo "$RESULTS" | grep -i "error"; then
handle_error "Failed to get query results: $RESULTS"
fi
echo "$RESULTS"
# Download results from S3
echo "Downloading query results from S3..."
S3_PATH=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.ResultConfiguration.OutputLocation" --output text 2>&1)
if echo "$S3_PATH" | grep -i "error"; then
handle_error "Failed to get S3 path for results: $S3_PATH"
fi
DOWNLOAD_RESULT=$(aws s3 cp "$S3_PATH" "./query-results.csv" 2>&1)
if echo "$DOWNLOAD_RESULT" | grep -i "error"; then
handle_error "Failed to download query results: $DOWNLOAD_RESULT"
fi
echo "Query results downloaded to query-results.csv"
# Step 4: Create a named query
echo "Step 4: Creating a named query..."
NAMED_QUERY_RESULT=$(aws athena create-named-query \
--name "OS Count Query" \
--description "Count of operating systems in CloudFront logs" \
--database "$DATABASE_NAME" \
--query-string "$QUERY" 2>&1)
if echo "$NAMED_QUERY_RESULT" | grep -i "error"; then
handle_error "Failed to create named query: $NAMED_QUERY_RESULT"
fi
NAMED_QUERY_ID=$(echo "$NAMED_QUERY_RESULT" | grep -o '"NamedQueryId": "[^"]*' | cut -d'"' -f4)
echo "Named query created with ID: $NAMED_QUERY_ID"
# List named queries
echo "Listing named queries..."
LIST_QUERIES_RESULT=$(aws athena list-named-queries 2>&1)
if echo "$LIST_QUERIES_RESULT" | grep -i "error"; then
handle_error "Failed to list named queries: $LIST_QUERIES_RESULT"
fi
echo "$LIST_QUERIES_RESULT"
# Get the named query details
echo "Getting named query details..."
GET_QUERY_RESULT=$(aws athena get-named-query --named-query-id "$NAMED_QUERY_ID" 2>&1)
if echo "$GET_QUERY_RESULT" | grep -i "error"; then
handle_error "Failed to get named query: $GET_QUERY_RESULT"
fi
echo "$GET_QUERY_RESULT"
# Execute the named query
echo "Executing the named query..."
QUERY_STRING=$(aws athena get-named-query --named-query-id "$NAMED_QUERY_ID" --query "NamedQuery.QueryString" --output text 2>&1)
if echo "$QUERY_STRING" | grep -i "error"; then
handle_error "Failed to get query string: $QUERY_STRING"
fi
EXEC_RESULT=$(aws athena start-query-execution \
--query-string "$QUERY_STRING" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$EXEC_RESULT" | grep -i "error"; then
handle_error "Failed to execute named query: $EXEC_RESULT"
fi
QUERY_ID=$(echo "$EXEC_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Named query execution ID: $QUERY_ID"
# Wait for named query to complete
echo "Waiting for named query execution to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Named query execution completed successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
handle_error "Named query execution failed with status: $QUERY_STATUS"
fi
echo "Named query execution in progress, status: $QUERY_STATUS"
sleep 2
done
# Summary of resources created
echo ""
echo "==========================================="
echo "RESOURCES CREATED"
echo "==========================================="
echo "- S3 Bucket: $S3_BUCKET"
echo "- Database: $DATABASE_NAME"
echo "- Table: $TABLE_NAME"
echo "- Named Query: $NAMED_QUERY_ID"
echo "- Query results saved to: query-results.csv"
echo "==========================================="
# Prompt for cleanup
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
echo "Starting cleanup..."
# Delete named query
echo "Deleting named query: $NAMED_QUERY_ID"
DELETE_QUERY_RESULT=$(aws athena delete-named-query --named-query-id "$NAMED_QUERY_ID" 2>&1)
if echo "$DELETE_QUERY_RESULT" | grep -i "error"; then
echo "Warning: Failed to delete named query: $DELETE_QUERY_RESULT"
else
echo "Named query deleted successfully."
fi
# Drop table
echo "Dropping table: $TABLE_NAME"
DROP_TABLE_RESULT=$(aws athena start-query-execution \
--query-string "DROP TABLE IF EXISTS $DATABASE_NAME.$TABLE_NAME" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$DROP_TABLE_RESULT" | grep -i "error"; then
echo "Warning: Failed to drop table: $DROP_TABLE_RESULT"
else
QUERY_ID=$(echo "$DROP_TABLE_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Waiting for table deletion to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Table dropped successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
echo "Warning: Table deletion failed with status: $QUERY_STATUS"
break
fi
echo "Table deletion in progress, status: $QUERY_STATUS"
sleep 2
done
fi
# Drop database
echo "Dropping database: $DATABASE_NAME"
DROP_DB_RESULT=$(aws athena start-query-execution \
--query-string "DROP DATABASE IF EXISTS $DATABASE_NAME" \
--result-configuration "OutputLocation=s3://$S3_BUCKET/output/" 2>&1)
if echo "$DROP_DB_RESULT" | grep -i "error"; then
echo "Warning: Failed to drop database: $DROP_DB_RESULT"
else
QUERY_ID=$(echo "$DROP_DB_RESULT" | grep -o '"QueryExecutionId": "[^"]*' | cut -d'"' -f4)
echo "Waiting for database deletion to complete..."
while true; do
QUERY_STATUS=$(aws athena get-query-execution --query-execution-id "$QUERY_ID" --query "QueryExecution.Status.State" --output text 2>&1)
if [ "$QUERY_STATUS" = "SUCCEEDED" ]; then
echo "Database dropped successfully."
break
elif [ "$QUERY_STATUS" = "FAILED" ] || [ "$QUERY_STATUS" = "CANCELLED" ]; then
echo "Warning: Database deletion failed with status: $QUERY_STATUS"
break
fi
echo "Database deletion in progress, status: $QUERY_STATUS"
sleep 2
done
fi
# Empty and delete S3 bucket
echo "Emptying S3 bucket: $S3_BUCKET"
EMPTY_BUCKET_RESULT=$(aws s3 rm "s3://$S3_BUCKET" --recursive 2>&1)
if echo "$EMPTY_BUCKET_RESULT" | grep -i "error"; then
echo "Warning: Failed to empty S3 bucket: $EMPTY_BUCKET_RESULT"
else
echo "S3 bucket emptied successfully."
fi
echo "Deleting S3 bucket: $S3_BUCKET"
DELETE_BUCKET_RESULT=$(aws s3 rb "s3://$S3_BUCKET" 2>&1)
if echo "$DELETE_BUCKET_RESULT" | grep -i "error"; then
echo "Warning: Failed to delete S3 bucket: $DELETE_BUCKET_RESULT"
else
echo "S3 bucket deleted successfully."
fi
echo "Cleanup completed."
else
echo "Cleanup skipped. Resources will remain in your AWS account."
fi
echo "Tutorial completed successfully!"
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [Cp](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Cp)
+ [CreateNamedQuery](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/CreateNamedQuery)
+ [DeleteNamedQuery](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/DeleteNamedQuery)
+ [GetNamedQuery](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/GetNamedQuery)
+ [GetQueryExecution](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/GetQueryExecution)
+ [GetQueryResults](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/GetQueryResults)
+ [ListDatabases](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/ListDatabases)
+ [ListNamedQueries](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/ListNamedQueries)
+ [ListTableMetadata](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/ListTableMetadata)
+ [Mb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Mb)
+ [Rb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rb)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
+ [StartQueryExecution](https://docs.aws.amazon.com/goto/aws-cli/athena-2017-05-18/StartQueryExecution)
------
# Getting started with Amazon EMR
The following code example shows how to:
+ Create an EC2 key pair
+ Set up storage and prepare your application
+ Clean up resources
------
#### [ Bash ]
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/037-emr-gs) repository.
```
#!/bin/bash
# EMR Getting Started Tutorial Script
# This script automates the steps in the Amazon EMR Getting Started tutorial
# Set up logging
LOG_FILE="emr-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "Starting Amazon EMR Getting Started Tutorial Script"
echo "Logging to $LOG_FILE"
# Function to handle errors
handle_error() {
echo "ERROR: $1"
echo "Resources created so far:"
if [ -n "$BUCKET_NAME" ]; then echo "- S3 Bucket: $BUCKET_NAME"; fi
if [ -n "$CLUSTER_ID" ]; then echo "- EMR Cluster: $CLUSTER_ID"; fi
echo "Attempting to clean up resources..."
cleanup
exit 1
}
# Function to clean up resources
cleanup() {
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "${CLEANUP_CHOICE,,}" == "y" ]]; then
echo "Starting cleanup process..."
# Terminate EMR cluster if it exists
if [ -n "$CLUSTER_ID" ]; then
echo "Terminating EMR cluster: $CLUSTER_ID"
aws emr terminate-clusters --cluster-ids "$CLUSTER_ID"
echo "Waiting for cluster to terminate..."
aws emr wait cluster-terminated --cluster-id "$CLUSTER_ID"
echo "Cluster terminated successfully."
fi
# Delete S3 bucket and contents if it exists
if [ -n "$BUCKET_NAME" ]; then
echo "Deleting S3 bucket contents: $BUCKET_NAME"
aws s3 rm "s3://$BUCKET_NAME" --recursive
echo "Deleting S3 bucket: $BUCKET_NAME"
aws s3 rb "s3://$BUCKET_NAME"
fi
echo "Cleanup completed."
else
echo "Cleanup skipped. Resources will remain in your AWS account."
echo "To avoid ongoing charges, remember to manually delete these resources."
fi
}
# Generate a random identifier for S3 bucket
RANDOM_ID=$(openssl rand -hex 6)
BUCKET_NAME="emr${RANDOM_ID}"
echo "Using bucket name: $BUCKET_NAME"
# Create S3 bucket
echo "Creating S3 bucket: $BUCKET_NAME"
aws s3 mb "s3://$BUCKET_NAME" || handle_error "Failed to create S3 bucket"
echo "S3 bucket created successfully."
# Create PySpark script
echo "Creating PySpark script: health_violations.py"
cat > health_violations.py << 'EOL'
import argparse
from pyspark.sql import SparkSession
def calculate_red_violations(data_source, output_uri):
"""
Processes sample food establishment inspection data and queries the data to find the top 10 establishments
with the most Red violations from 2006 to 2020.
:param data_source: The URI of your food establishment data CSV, such as 's3://emr-tutorial-bucket/food-establishment-data.csv'.
:param output_uri: The URI where output is written, such as 's3://emr-tutorial-bucket/restaurant_violation_results'.
"""
with SparkSession.builder.appName("Calculate Red Health Violations").getOrCreate() as spark:
# Load the restaurant violation CSV data
if data_source is not None:
restaurants_df = spark.read.option("header", "true").csv(data_source)
# Create an in-memory DataFrame to query
restaurants_df.createOrReplaceTempView("restaurant_violations")
# Create a DataFrame of the top 10 restaurants with the most Red violations
top_red_violation_restaurants = spark.sql("""SELECT name, count(*) AS total_red_violations
FROM restaurant_violations
WHERE violation_type = 'RED'
GROUP BY name
ORDER BY total_red_violations DESC LIMIT 10""")
# Write the results to the specified output URI
top_red_violation_restaurants.write.option("header", "true").mode("overwrite").csv(output_uri)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
'--data_source', help="The URI for you CSV restaurant data, like an S3 bucket location.")
parser.add_argument(
'--output_uri', help="The URI where output is saved, like an S3 bucket location.")
args = parser.parse_args()
calculate_red_violations(args.data_source, args.output_uri)
EOL
# Upload PySpark script to S3
echo "Uploading PySpark script to S3"
aws s3 cp health_violations.py "s3://$BUCKET_NAME/" || handle_error "Failed to upload PySpark script"
echo "PySpark script uploaded successfully."
# Download and prepare sample data
echo "Downloading sample data"
curl -o food_establishment_data.zip https://docs.aws.amazon.com/emr/latest/ManagementGuide/samples/food_establishment_data.zip || handle_error "Failed to download sample data"
unzip -o food_establishment_data.zip || handle_error "Failed to unzip sample data"
echo "Sample data downloaded and extracted successfully."
# Upload sample data to S3
echo "Uploading sample data to S3"
aws s3 cp food_establishment_data.csv "s3://$BUCKET_NAME/" || handle_error "Failed to upload sample data"
echo "Sample data uploaded successfully."
# Create IAM default roles for EMR
echo "Creating IAM default roles for EMR"
aws emr create-default-roles || handle_error "Failed to create default roles"
echo "IAM default roles created successfully."
# Check if EC2 key pair exists
echo "Checking for EC2 key pair"
KEY_PAIRS=$(aws ec2 describe-key-pairs --query "KeyPairs[*].KeyName" --output text)
if [ -z "$KEY_PAIRS" ]; then
echo "No EC2 key pairs found. Creating a new key pair..."
KEY_NAME="emr-tutorial-key-$RANDOM_ID"
aws ec2 create-key-pair --key-name "$KEY_NAME" --query "KeyMaterial" --output text > "${KEY_NAME}.pem"
chmod 400 "${KEY_NAME}.pem"
echo "Created new key pair: $KEY_NAME"
else
# Use the first available key pair
KEY_NAME=$(echo "$KEY_PAIRS" | awk '{print $1}')
echo "Using existing key pair: $KEY_NAME"
fi
# Launch EMR cluster
echo "Launching EMR cluster with Spark"
CLUSTER_RESPONSE=$(aws emr create-cluster \
--name "EMR Tutorial Cluster" \
--release-label emr-6.10.0 \
--applications Name=Spark \
--ec2-attributes KeyName="$KEY_NAME" \
--instance-type m5.xlarge \
--instance-count 3 \
--use-default-roles \
--log-uri "s3://$BUCKET_NAME/logs/")
# Check for errors in the response
if echo "$CLUSTER_RESPONSE" | grep -i "error" > /dev/null; then
handle_error "Failed to create EMR cluster: $CLUSTER_RESPONSE"
fi
# Extract cluster ID
CLUSTER_ID=$(echo "$CLUSTER_RESPONSE" | grep -o '"ClusterId": "[^"]*' | cut -d'"' -f4)
if [ -z "$CLUSTER_ID" ]; then
handle_error "Failed to extract cluster ID from response"
fi
echo "EMR cluster created with ID: $CLUSTER_ID"
# Wait for cluster to be ready
echo "Waiting for cluster to be ready (this may take several minutes)..."
aws emr wait cluster-running --cluster-id "$CLUSTER_ID" || handle_error "Cluster failed to reach running state"
# Check if cluster is in WAITING state
CLUSTER_STATE=$(aws emr describe-cluster --cluster-id "$CLUSTER_ID" --query "Cluster.Status.State" --output text)
if [ "$CLUSTER_STATE" != "WAITING" ]; then
echo "Waiting for cluster to reach WAITING state..."
while [ "$CLUSTER_STATE" != "WAITING" ]; do
sleep 30
CLUSTER_STATE=$(aws emr describe-cluster --cluster-id "$CLUSTER_ID" --query "Cluster.Status.State" --output text)
echo "Current cluster state: $CLUSTER_STATE"
# Check for error states
if [[ "$CLUSTER_STATE" == "TERMINATED_WITH_ERRORS" || "$CLUSTER_STATE" == "TERMINATED" ]]; then
handle_error "Cluster entered error state: $CLUSTER_STATE"
fi
done
fi
echo "Cluster is now in WAITING state and ready to accept work."
# Submit Spark application as a step
echo "Submitting Spark application as a step"
STEP_RESPONSE=$(aws emr add-steps \
--cluster-id "$CLUSTER_ID" \
--steps Type=Spark,Name="Health Violations Analysis",ActionOnFailure=CONTINUE,Args=["s3://$BUCKET_NAME/health_violations.py","--data_source","s3://$BUCKET_NAME/food_establishment_data.csv","--output_uri","s3://$BUCKET_NAME/results/"])
# Check for errors in the response
if echo "$STEP_RESPONSE" | grep -i "error" > /dev/null; then
handle_error "Failed to submit step: $STEP_RESPONSE"
fi
# FIXED: Check if jq is available before using it
# Extract step ID using the appropriate method based on available tools
if command -v jq &> /dev/null; then
# Use jq if available
echo "Using jq to parse JSON response"
STEP_ID=$(echo "$STEP_RESPONSE" | jq -r '.StepIds[0]')
else
# Fallback to grep/awk if jq is not available
echo "jq not found, using grep for parsing"
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*"' | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
# Another fallback method
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*' | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
# One more attempt with a different pattern
STEP_ID=$(echo "$STEP_RESPONSE" | grep -o 's-[A-Z0-9]*')
if [ -z "$STEP_ID" ]; then
echo "Full step response: $STEP_RESPONSE"
handle_error "Failed to extract step ID from response"
fi
fi
fi
fi
if [ -z "$STEP_ID" ] || [ "$STEP_ID" == "null" ]; then
echo "Full step response: $STEP_RESPONSE"
handle_error "Failed to extract valid step ID from response"
fi
echo "Step submitted with ID: $STEP_ID"
# Wait for step to complete
echo "Waiting for step to complete (this may take several minutes)..."
aws emr wait step-complete --cluster-id "$CLUSTER_ID" --step-id "$STEP_ID" || handle_error "Step failed to complete"
# Check step status
STEP_STATE=$(aws emr describe-step --cluster-id "$CLUSTER_ID" --step-id "$STEP_ID" --query "Step.Status.State" --output text)
if [ "$STEP_STATE" != "COMPLETED" ]; then
handle_error "Step did not complete successfully. Final state: $STEP_STATE"
fi
echo "Step completed successfully."
# View results
echo "Listing output files in S3"
aws s3 ls "s3://$BUCKET_NAME/results/" || handle_error "Failed to list output files"
# Download results
echo "Downloading results file"
RESULT_FILE=$(aws s3 ls "s3://$BUCKET_NAME/results/" | grep -o "part-[0-9]*.csv" | head -1)
if [ -z "$RESULT_FILE" ]; then
echo "No result file found with pattern 'part-[0-9]*.csv'. Trying to find any CSV file..."
RESULT_FILE=$(aws s3 ls "s3://$BUCKET_NAME/results/" | grep -o "part-.*\.csv" | head -1)
if [ -z "$RESULT_FILE" ]; then
echo "Listing all files in results directory:"
aws s3 ls "s3://$BUCKET_NAME/results/"
handle_error "No result file found in the output directory"
fi
fi
aws s3 cp "s3://$BUCKET_NAME/results/$RESULT_FILE" ./results.csv || handle_error "Failed to download results file"
echo "Results downloaded to results.csv"
echo "Top 10 establishments with the most red violations:"
cat results.csv
# Display SSH connection information
echo ""
echo "To connect to the cluster via SSH, use the following command:"
echo "aws emr ssh --cluster-id $CLUSTER_ID --key-pair-file ${KEY_NAME}.pem"
# Display summary of created resources
echo ""
echo "==========================================="
echo "RESOURCES CREATED"
echo "==========================================="
echo "- S3 Bucket: $BUCKET_NAME"
echo "- EMR Cluster: $CLUSTER_ID"
echo "- Results file: results.csv"
if [ -f "${KEY_NAME}.pem" ]; then
echo "- EC2 Key Pair: $KEY_NAME (saved to ${KEY_NAME}.pem)"
fi
# Offer to clean up resources
cleanup
echo "Script completed successfully."
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AddSteps](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/AddSteps)
+ [Cp](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Cp)
+ [CreateCluster](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/CreateCluster)
+ [CreateDefaultRoles](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/CreateDefaultRoles)
+ [CreateKeyPair](https://docs.aws.amazon.com/goto/aws-cli/ec2-2016-11-15/CreateKeyPair)
+ [DescribeCluster](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/DescribeCluster)
+ [DescribeKeyPairs](https://docs.aws.amazon.com/goto/aws-cli/ec2-2016-11-15/DescribeKeyPairs)
+ [DescribeStep](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/DescribeStep)
+ [Ls](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Ls)
+ [Mb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Mb)
+ [Rb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rb)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
+ [Ssh](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/Ssh)
+ [TerminateClusters](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/TerminateClusters)
+ [Wait](https://docs.aws.amazon.com/goto/aws-cli/elasticmapreduce-2009-03-31/Wait)
------
# Getting started with Amazon S3
The following code example shows how to:
+ Create your first S3 bucket
+ Upload an object
+ Enable versioning
+ Configure default encryption
+ Add tags to your bucket
+ List objects and versions
+ Clean up resources
------
#### [ Bash ]
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/003-s3-gettingstarted) repository.
```
#!/bin/bash
# S3 Getting Started - Create a bucket, upload and download objects, copy to a
# folder prefix, enable versioning, configure encryption and public access
# blocking, tag the bucket, list objects and versions, and clean up.
set -eE
# ============================================================================
# Prerequisites check
# ============================================================================
CONFIGURED_REGION=$(aws configure get region 2>/dev/null || true)
if [ -z "$CONFIGURED_REGION" ] && [ -z "$AWS_DEFAULT_REGION" ] && [ -z "$AWS_REGION" ]; then
echo "ERROR: No AWS region configured. Run 'aws configure' or set AWS_DEFAULT_REGION."
exit 1
fi
# ============================================================================
# Setup: logging, temp directory, resource tracking
# ============================================================================
UNIQUE_ID=$(cat /dev/urandom | tr -dc 'a-f0-9' | fold -w 12 | head -n 1)
BUCKET_NAME="s3api-${UNIQUE_ID}"
TEMP_DIR=$(mktemp -d)
LOG_FILE="${TEMP_DIR}/s3-gettingstarted.log"
CREATED_RESOURCES=()
exec > >(tee -a "$LOG_FILE") 2>&1
echo "============================================"
echo "S3 Getting Started"
echo "============================================"
echo "Bucket name: ${BUCKET_NAME}"
echo "Temp directory: ${TEMP_DIR}"
echo "Log file: ${LOG_FILE}"
echo ""
# ============================================================================
# Error handling and cleanup functions
# ============================================================================
cleanup() {
echo ""
echo "============================================"
echo "CLEANUP"
echo "============================================"
# Delete all object versions and delete markers
echo "Listing all object versions in bucket..."
VERSIONS_OUTPUT=$(aws s3api list-object-versions \
--bucket "$BUCKET_NAME" \
--query "Versions[].{Key:Key,VersionId:VersionId}" \
--output text 2>&1) || true
if [ -n "$VERSIONS_OUTPUT" ] && [ "$VERSIONS_OUTPUT" != "None" ]; then
while IFS=$'\t' read -r KEY VERSION_ID; do
if [ -n "$KEY" ] && [ "$KEY" != "None" ]; then
echo "Deleting version: ${KEY} (${VERSION_ID})"
aws s3api delete-object \
--bucket "$BUCKET_NAME" \
--key "$KEY" \
--version-id "$VERSION_ID" 2>&1 || echo "WARNING: Failed to delete version ${KEY} (${VERSION_ID})"
fi
done <<< "$VERSIONS_OUTPUT"
fi
DELETE_MARKERS_OUTPUT=$(aws s3api list-object-versions \
--bucket "$BUCKET_NAME" \
--query "DeleteMarkers[].{Key:Key,VersionId:VersionId}" \
--output text 2>&1) || true
if [ -n "$DELETE_MARKERS_OUTPUT" ] && [ "$DELETE_MARKERS_OUTPUT" != "None" ]; then
while IFS=$'\t' read -r KEY VERSION_ID; do
if [ -n "$KEY" ] && [ "$KEY" != "None" ]; then
echo "Deleting delete marker: ${KEY} (${VERSION_ID})"
aws s3api delete-object \
--bucket "$BUCKET_NAME" \
--key "$KEY" \
--version-id "$VERSION_ID" 2>&1 || echo "WARNING: Failed to delete marker ${KEY} (${VERSION_ID})"
fi
done <<< "$DELETE_MARKERS_OUTPUT"
fi
echo "Deleting bucket: ${BUCKET_NAME}"
aws s3api delete-bucket --bucket "$BUCKET_NAME" 2>&1 || echo "WARNING: Failed to delete bucket ${BUCKET_NAME}"
echo ""
echo "Cleaning up temp directory: ${TEMP_DIR}"
rm -rf "$TEMP_DIR"
echo ""
echo "Cleanup complete."
}
handle_error() {
echo ""
echo "============================================"
echo "ERROR on $1"
echo "============================================"
echo ""
echo "Resources created before error:"
for RESOURCE in "${CREATED_RESOURCES[@]}"; do
echo " - ${RESOURCE}"
done
echo ""
echo "Attempting cleanup..."
cleanup
exit 1
}
trap 'handle_error "line $LINENO"' ERR
# ============================================================================
# Step 1: Create a bucket
# ============================================================================
echo "Step 1: Creating bucket ${BUCKET_NAME}..."
# CreateBucket requires LocationConstraint for all regions except us-east-1
REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-${CONFIGURED_REGION}}}"
if [ "$REGION" = "us-east-1" ]; then
CREATE_OUTPUT=$(aws s3api create-bucket \
--bucket "$BUCKET_NAME" 2>&1)
else
CREATE_OUTPUT=$(aws s3api create-bucket \
--bucket "$BUCKET_NAME" \
--create-bucket-configuration LocationConstraint="$REGION" 2>&1)
fi
echo "$CREATE_OUTPUT"
CREATED_RESOURCES+=("s3:bucket:${BUCKET_NAME}")
echo "Bucket created."
echo ""
# ============================================================================
# Step 2: Upload a sample text file
# ============================================================================
echo "Step 2: Uploading a sample text file..."
SAMPLE_FILE="${TEMP_DIR}/sample.txt"
echo "Hello, Amazon S3! This is a sample file for the getting started tutorial." > "$SAMPLE_FILE"
UPLOAD_OUTPUT=$(aws s3api put-object \
--bucket "$BUCKET_NAME" \
--key "sample.txt" \
--body "$SAMPLE_FILE" 2>&1)
echo "$UPLOAD_OUTPUT"
echo "File uploaded."
echo ""
# ============================================================================
# Step 3: Download the object
# ============================================================================
echo "Step 3: Downloading the object..."
DOWNLOAD_FILE="${TEMP_DIR}/downloaded-sample.txt"
aws s3api get-object \
--bucket "$BUCKET_NAME" \
--key "sample.txt" \
"$DOWNLOAD_FILE" 2>&1
echo "Downloaded to: ${DOWNLOAD_FILE}"
echo "Contents:"
cat "$DOWNLOAD_FILE"
echo ""
# ============================================================================
# Step 4: Copy the object to a folder prefix
# ============================================================================
echo "Step 4: Copying object to a folder prefix..."
COPY_OUTPUT=$(aws s3api copy-object \
--bucket "$BUCKET_NAME" \
--copy-source "${BUCKET_NAME}/sample.txt" \
--key "backup/sample.txt" 2>&1)
echo "$COPY_OUTPUT"
echo "Object copied to backup/sample.txt."
echo ""
# ============================================================================
# Step 5: Enable versioning and upload a second version
# ============================================================================
echo "Step 5: Enabling versioning..."
VERSIONING_OUTPUT=$(aws s3api put-bucket-versioning \
--bucket "$BUCKET_NAME" \
--versioning-configuration Status=Enabled 2>&1)
echo "$VERSIONING_OUTPUT"
echo "Versioning enabled."
echo "Uploading a second version of sample.txt..."
echo "Hello, Amazon S3! This is version 2 of the sample file." > "$SAMPLE_FILE"
UPLOAD_V2_OUTPUT=$(aws s3api put-object \
--bucket "$BUCKET_NAME" \
--key "sample.txt" \
--body "$SAMPLE_FILE" 2>&1)
echo "$UPLOAD_V2_OUTPUT"
echo "Second version uploaded."
echo ""
# ============================================================================
# Step 6: Configure SSE-S3 encryption
# ============================================================================
echo "Step 6: Configuring SSE-S3 default encryption..."
ENCRYPTION_OUTPUT=$(aws s3api put-bucket-encryption \
--bucket "$BUCKET_NAME" \
--server-side-encryption-configuration '{
"Rules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "AES256"
},
"BucketKeyEnabled": true
}
]
}' 2>&1)
echo "$ENCRYPTION_OUTPUT"
echo "SSE-S3 encryption configured."
echo ""
# ============================================================================
# Step 7: Block all public access
# ============================================================================
echo "Step 7: Blocking all public access..."
PUBLIC_ACCESS_OUTPUT=$(aws s3api put-public-access-block \
--bucket "$BUCKET_NAME" \
--public-access-block-configuration '{
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}' 2>&1)
echo "$PUBLIC_ACCESS_OUTPUT"
echo "Public access blocked."
echo ""
# ============================================================================
# Step 8: Tag the bucket
# ============================================================================
echo "Step 8: Tagging the bucket..."
TAG_OUTPUT=$(aws s3api put-bucket-tagging \
--bucket "$BUCKET_NAME" \
--tagging '{
"TagSet": [
{
"Key": "Environment",
"Value": "Tutorial"
},
{
"Key": "Project",
"Value": "S3-GettingStarted"
}
]
}' 2>&1)
echo "$TAG_OUTPUT"
echo "Bucket tagged."
echo "Verifying tags..."
GET_TAGS_OUTPUT=$(aws s3api get-bucket-tagging \
--bucket "$BUCKET_NAME" 2>&1)
echo "$GET_TAGS_OUTPUT"
echo ""
# ============================================================================
# Step 9: List objects and versions
# ============================================================================
echo "Step 9: Listing objects..."
LIST_OUTPUT=$(aws s3api list-objects-v2 \
--bucket "$BUCKET_NAME" 2>&1)
echo "$LIST_OUTPUT"
echo ""
echo "Listing object versions..."
VERSIONS_LIST=$(aws s3api list-object-versions \
--bucket "$BUCKET_NAME" 2>&1)
echo "$VERSIONS_LIST"
echo ""
# ============================================================================
# Step 10: Cleanup
# ============================================================================
echo ""
echo "============================================"
echo "TUTORIAL COMPLETE"
echo "============================================"
echo ""
echo "Resources created:"
for RESOURCE in "${CREATED_RESOURCES[@]}"; do
echo " - ${RESOURCE}"
done
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
cleanup
else
echo ""
echo "Resources were NOT deleted. To clean up manually, run:"
echo ""
echo " # Delete all object versions"
echo " aws s3api list-object-versions --bucket ${BUCKET_NAME} --query 'Versions[].{Key:Key,VersionId:VersionId}' --output text | while IFS=\$'\\t' read -r KEY VID; do aws s3api delete-object --bucket ${BUCKET_NAME} --key \"\$KEY\" --version-id \"\$VID\"; done"
echo ""
echo " # Delete all delete markers"
echo " aws s3api list-object-versions --bucket ${BUCKET_NAME} --query 'DeleteMarkers[].{Key:Key,VersionId:VersionId}' --output text | while IFS=\$'\\t' read -r KEY VID; do aws s3api delete-object --bucket ${BUCKET_NAME} --key \"\$KEY\" --version-id \"\$VID\"; done"
echo ""
echo " # Delete the bucket"
echo " aws s3api delete-bucket --bucket ${BUCKET_NAME}"
echo ""
echo " # Remove temp directory"
echo " rm -rf ${TEMP_DIR}"
fi
echo ""
echo "Done."
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [CopyObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CopyObject)
+ [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket)
+ [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket)
+ [DeleteObjects](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteObjects)
+ [GetObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/GetObject)
+ [HeadObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/HeadObject)
+ [ListObjectVersions](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/ListObjectVersions)
+ [ListObjectsV2](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/ListObjectsV2)
+ [PutBucketEncryption](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutBucketEncryption)
+ [PutBucketTagging](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutBucketTagging)
+ [PutBucketVersioning](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutBucketVersioning)
+ [PutObject](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutObject)
+ [PutPublicAccessBlock](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutPublicAccessBlock)
------
# Getting started with Amazon SageMaker Feature Store
The following code example shows how to:
+ Set up IAM permissions
+ Create a SageMaker execution role
+ Create feature groups
+ Clean up resources
------
#### [ Bash ]
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/028-sagemaker-featurestore) repository.
```
#!/bin/bash
# Amazon SageMaker Feature Store Tutorial Script - Version 3
# This script demonstrates how to use Amazon SageMaker Feature Store with AWS CLI
# Setup logging
LOG_FILE="sagemaker-featurestore-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "Starting SageMaker Feature Store tutorial script at $(date)"
echo "All commands and outputs will be logged to $LOG_FILE"
echo ""
# Track created resources for cleanup
CREATED_RESOURCES=()
# Function to handle errors
handle_error() {
echo "ERROR: $1"
echo "Attempting to clean up resources..."
cleanup_resources
exit 1
}
# Function to check command status
check_status() {
if echo "$1" | grep -i "error" > /dev/null; then
handle_error "$1"
fi
}
# Function to wait for feature group to be created
wait_for_feature_group() {
local feature_group_name=$1
local status="Creating"
echo "Waiting for feature group ${feature_group_name} to be created..."
while [ "$status" = "Creating" ]; do
sleep 5
status=$(aws sagemaker describe-feature-group \
--feature-group-name "${feature_group_name}" \
--query 'FeatureGroupStatus' \
--output text)
echo "Current status: ${status}"
if [ "$status" = "Failed" ]; then
handle_error "Feature group ${feature_group_name} creation failed"
fi
done
echo "Feature group ${feature_group_name} is now ${status}"
}
# Function to clean up resources
cleanup_resources() {
echo "Cleaning up resources..."
# Clean up in reverse order
for ((i=${#CREATED_RESOURCES[@]}-1; i>=0; i--)); do
resource="${CREATED_RESOURCES[$i]}"
resource_type=$(echo "$resource" | cut -d: -f1)
resource_name=$(echo "$resource" | cut -d: -f2)
echo "Deleting $resource_type: $resource_name"
case "$resource_type" in
"FeatureGroup")
aws sagemaker delete-feature-group --feature-group-name "$resource_name"
;;
"S3Bucket")
echo "Emptying S3 bucket: $resource_name"
aws s3 rm "s3://$resource_name" --recursive 2>/dev/null
echo "Deleting S3 bucket: $resource_name"
aws s3api delete-bucket --bucket "$resource_name" 2>/dev/null
;;
"IAMRole")
echo "Detaching policies from role: $resource_name"
aws iam detach-role-policy --role-name "$resource_name" --policy-arn "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess" 2>/dev/null
aws iam detach-role-policy --role-name "$resource_name" --policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess" 2>/dev/null
echo "Deleting IAM role: $resource_name"
aws iam delete-role --role-name "$resource_name" 2>/dev/null
;;
*)
echo "Unknown resource type: $resource_type"
;;
esac
done
}
# Function to create SageMaker execution role
create_sagemaker_role() {
local role_name="SageMakerFeatureStoreRole-$(openssl rand -hex 4)"
echo "Creating SageMaker execution role: $role_name" >&2
# Create trust policy document
local trust_policy='{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
# Create the role
local role_result=$(aws iam create-role \
--role-name "$role_name" \
--assume-role-policy-document "$trust_policy" \
--description "SageMaker execution role for Feature Store tutorial" 2>&1)
if echo "$role_result" | grep -i "error" > /dev/null; then
handle_error "Failed to create IAM role: $role_result"
fi
echo "Role created successfully" >&2
CREATED_RESOURCES+=("IAMRole:$role_name")
# Attach necessary policies
echo "Attaching policies to role..." >&2
# SageMaker execution policy
local policy1_result=$(aws iam attach-role-policy \
--role-name "$role_name" \
--policy-arn "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess" 2>&1)
if echo "$policy1_result" | grep -i "error" > /dev/null; then
handle_error "Failed to attach SageMaker policy: $policy1_result"
fi
# S3 access policy
local policy2_result=$(aws iam attach-role-policy \
--role-name "$role_name" \
--policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess" 2>&1)
if echo "$policy2_result" | grep -i "error" > /dev/null; then
handle_error "Failed to attach S3 policy: $policy2_result"
fi
# Get account ID for role ARN
local account_id=$(aws sts get-caller-identity --query Account --output text)
local role_arn="arn:aws:iam::${account_id}:role/${role_name}"
echo "Role ARN: $role_arn" >&2
echo "Waiting 10 seconds for role to propagate..." >&2
sleep 10
# Return only the role ARN to stdout
echo "$role_arn"
}
# Handle SageMaker execution role
ROLE_ARN=""
if [ -z "$1" ]; then
echo "Creating SageMaker execution role automatically..."
ROLE_ARN=$(create_sagemaker_role)
if [ -z "$ROLE_ARN" ]; then
handle_error "Failed to create SageMaker execution role"
fi
else
ROLE_ARN="$1"
# Validate the role ARN
ROLE_NAME=$(echo "$ROLE_ARN" | sed 's/.*role\///')
ROLE_CHECK=$(aws iam get-role --role-name "$ROLE_NAME" 2>&1)
if echo "$ROLE_CHECK" | grep -i "error" > /dev/null; then
echo "Creating a new role automatically..."
ROLE_ARN=$(create_sagemaker_role)
if [ -z "$ROLE_ARN" ]; then
handle_error "Failed to create SageMaker execution role"
fi
fi
fi
# Handle cleanup option
AUTO_CLEANUP=""
if [ -n "$2" ]; then
AUTO_CLEANUP="$2"
fi
# Generate a random identifier for resource names
RANDOM_ID=$(openssl rand -hex 4)
echo "Using random identifier: $RANDOM_ID"
# Set variables
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
check_status "$ACCOUNT_ID"
echo "Account ID: $ACCOUNT_ID"
# Get current region
REGION=$(aws configure get region)
if [ -z "$REGION" ]; then
REGION="us-east-1"
echo "No default region configured, using: $REGION"
else
echo "Using region: $REGION"
fi
S3_BUCKET_NAME="sagemaker-featurestore-${RANDOM_ID}-${ACCOUNT_ID}"
PREFIX="featurestore-tutorial"
CURRENT_TIME=$(date +%s)
echo "Creating S3 bucket: $S3_BUCKET_NAME"
# Create bucket in current region
if [ "$REGION" = "us-east-1" ]; then
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME" \
--region "$REGION" 2>&1)
else
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME" \
--region "$REGION" \
--create-bucket-configuration LocationConstraint="$REGION" 2>&1)
fi
if echo "$BUCKET_RESULT" | grep -i "error" > /dev/null; then
echo "Failed to create S3 bucket: $BUCKET_RESULT"
exit 1
fi
echo "$BUCKET_RESULT"
CREATED_RESOURCES+=("S3Bucket:$S3_BUCKET_NAME")
# Block public access to the bucket
BLOCK_RESULT=$(aws s3api put-public-access-block \
--bucket "$S3_BUCKET_NAME" \
--public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" 2>&1)
if echo "$BLOCK_RESULT" | grep -i "error" > /dev/null; then
echo "Failed to block public access to S3 bucket: $BLOCK_RESULT"
cleanup_resources
exit 1
fi
# Create feature groups
echo "Creating feature groups..."
# Create customers feature group
CUSTOMERS_FEATURE_GROUP_NAME="customers-feature-group-${RANDOM_ID}"
echo "Creating customers feature group: $CUSTOMERS_FEATURE_GROUP_NAME"
CUSTOMERS_RESPONSE=$(aws sagemaker create-feature-group \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record-identifier-feature-name "customer_id" \
--event-time-feature-name "EventTime" \
--feature-definitions '[
{"FeatureName": "customer_id", "FeatureType": "Integral"},
{"FeatureName": "name", "FeatureType": "String"},
{"FeatureName": "age", "FeatureType": "Integral"},
{"FeatureName": "address", "FeatureType": "String"},
{"FeatureName": "membership_type", "FeatureType": "String"},
{"FeatureName": "EventTime", "FeatureType": "Fractional"}
]' \
--online-store-config '{"EnableOnlineStore": true}' \
--offline-store-config '{
"S3StorageConfig": {
"S3Uri": "s3://'${S3_BUCKET_NAME}'/'${PREFIX}'"
},
"DisableGlueTableCreation": false
}' \
--role-arn "$ROLE_ARN" 2>&1)
if echo "$CUSTOMERS_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to create customers feature group: $CUSTOMERS_RESPONSE"
cleanup_resources
exit 1
fi
echo "$CUSTOMERS_RESPONSE"
CREATED_RESOURCES+=("FeatureGroup:$CUSTOMERS_FEATURE_GROUP_NAME")
# Create orders feature group
ORDERS_FEATURE_GROUP_NAME="orders-feature-group-${RANDOM_ID}"
echo "Creating orders feature group: $ORDERS_FEATURE_GROUP_NAME"
ORDERS_RESPONSE=$(aws sagemaker create-feature-group \
--feature-group-name "$ORDERS_FEATURE_GROUP_NAME" \
--record-identifier-feature-name "customer_id" \
--event-time-feature-name "EventTime" \
--feature-definitions '[
{"FeatureName": "customer_id", "FeatureType": "Integral"},
{"FeatureName": "order_id", "FeatureType": "String"},
{"FeatureName": "order_date", "FeatureType": "String"},
{"FeatureName": "product", "FeatureType": "String"},
{"FeatureName": "quantity", "FeatureType": "Integral"},
{"FeatureName": "amount", "FeatureType": "Fractional"},
{"FeatureName": "EventTime", "FeatureType": "Fractional"}
]' \
--online-store-config '{"EnableOnlineStore": true}' \
--offline-store-config '{
"S3StorageConfig": {
"S3Uri": "s3://'${S3_BUCKET_NAME}'/'${PREFIX}'"
},
"DisableGlueTableCreation": false
}' \
--role-arn "$ROLE_ARN" 2>&1)
if echo "$ORDERS_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to create orders feature group: $ORDERS_RESPONSE"
cleanup_resources
exit 1
fi
echo "$ORDERS_RESPONSE"
CREATED_RESOURCES+=("FeatureGroup:$ORDERS_FEATURE_GROUP_NAME")
# Wait for feature groups to be created
wait_for_feature_group "$CUSTOMERS_FEATURE_GROUP_NAME"
wait_for_feature_group "$ORDERS_FEATURE_GROUP_NAME"
# Ingest data into feature groups
echo "Ingesting data into feature groups..."
# Ingest customer data
echo "Ingesting customer data..."
CUSTOMER1_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "573291"},
{"FeatureName": "name", "ValueAsString": "John Doe"},
{"FeatureName": "age", "ValueAsString": "35"},
{"FeatureName": "address", "ValueAsString": "123 Main St"},
{"FeatureName": "membership_type", "ValueAsString": "premium"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$CUSTOMER1_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest customer 1 data: $CUSTOMER1_RESPONSE"
cleanup_resources
exit 1
fi
echo "$CUSTOMER1_RESPONSE"
CUSTOMER2_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "109382"},
{"FeatureName": "name", "ValueAsString": "Jane Smith"},
{"FeatureName": "age", "ValueAsString": "28"},
{"FeatureName": "address", "ValueAsString": "456 Oak Ave"},
{"FeatureName": "membership_type", "ValueAsString": "standard"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$CUSTOMER2_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest customer 2 data: $CUSTOMER2_RESPONSE"
cleanup_resources
exit 1
fi
echo "$CUSTOMER2_RESPONSE"
# Ingest order data
echo "Ingesting order data..."
ORDER1_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$ORDERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "573291"},
{"FeatureName": "order_id", "ValueAsString": "ORD-001"},
{"FeatureName": "order_date", "ValueAsString": "2023-01-15"},
{"FeatureName": "product", "ValueAsString": "Laptop"},
{"FeatureName": "quantity", "ValueAsString": "1"},
{"FeatureName": "amount", "ValueAsString": "1299.99"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$ORDER1_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest order 1 data: $ORDER1_RESPONSE"
cleanup_resources
exit 1
fi
echo "$ORDER1_RESPONSE"
ORDER2_RESPONSE=$(aws sagemaker-featurestore-runtime put-record \
--feature-group-name "$ORDERS_FEATURE_GROUP_NAME" \
--record '[
{"FeatureName": "customer_id", "ValueAsString": "109382"},
{"FeatureName": "order_id", "ValueAsString": "ORD-002"},
{"FeatureName": "order_date", "ValueAsString": "2023-01-20"},
{"FeatureName": "product", "ValueAsString": "Smartphone"},
{"FeatureName": "quantity", "ValueAsString": "1"},
{"FeatureName": "amount", "ValueAsString": "899.99"},
{"FeatureName": "EventTime", "ValueAsString": "'${CURRENT_TIME}'"}
]' 2>&1)
if echo "$ORDER2_RESPONSE" | grep -i "error" > /dev/null; then
echo "Failed to ingest order 2 data: $ORDER2_RESPONSE"
cleanup_resources
exit 1
fi
echo "$ORDER2_RESPONSE"
# Retrieve records from feature groups
echo "Retrieving records from feature groups..."
# Get a single customer record
echo "Getting customer record with ID 573291:"
CUSTOMER_RECORD=$(aws sagemaker-featurestore-runtime get-record \
--feature-group-name "$CUSTOMERS_FEATURE_GROUP_NAME" \
--record-identifier-value-as-string "573291" 2>&1)
if echo "$CUSTOMER_RECORD" | grep -i "error" > /dev/null; then
echo "Failed to get customer record: $CUSTOMER_RECORD"
cleanup_resources
exit 1
fi
echo "$CUSTOMER_RECORD"
# Get multiple records using batch-get-record
echo "Getting multiple records using batch-get-record:"
BATCH_RECORDS=$(aws sagemaker-featurestore-runtime batch-get-record \
--identifiers '[
{
"FeatureGroupName": "'${CUSTOMERS_FEATURE_GROUP_NAME}'",
"RecordIdentifiersValueAsString": ["573291", "109382"]
},
{
"FeatureGroupName": "'${ORDERS_FEATURE_GROUP_NAME}'",
"RecordIdentifiersValueAsString": ["573291", "109382"]
}
]' 2>&1)
if echo "$BATCH_RECORDS" | grep -i "error" > /dev/null && ! echo "$BATCH_RECORDS" | grep -i "Records" > /dev/null; then
echo "Failed to get batch records: $BATCH_RECORDS"
cleanup_resources
exit 1
fi
echo "$BATCH_RECORDS"
# List feature groups
echo "Listing feature groups:"
FEATURE_GROUPS=$(aws sagemaker list-feature-groups 2>&1)
if echo "$FEATURE_GROUPS" | grep -i "error" > /dev/null; then
echo "Failed to list feature groups: $FEATURE_GROUPS"
cleanup_resources
exit 1
fi
echo "$FEATURE_GROUPS"
# Display summary of created resources
echo ""
echo "==========================================="
echo "TUTORIAL COMPLETED SUCCESSFULLY!"
echo "==========================================="
echo "Resources created:"
echo "- S3 Bucket: $S3_BUCKET_NAME"
echo "- Customers Feature Group: $CUSTOMERS_FEATURE_GROUP_NAME"
echo "- Orders Feature Group: $ORDERS_FEATURE_GROUP_NAME"
if [[ " ${CREATED_RESOURCES[@]} " =~ " IAMRole:" ]]; then
echo "- IAM Role: $(echo "${CREATED_RESOURCES[@]}" | grep -o 'IAMRole:[^[:space:]]*' | cut -d: -f2)"
fi
echo ""
echo "You can now:"
echo "1. View your feature groups in the SageMaker console"
echo "2. Query the offline store using Amazon Athena"
echo "3. Use the feature groups in your ML workflows"
echo "==========================================="
echo ""
# Handle cleanup
if [ "$AUTO_CLEANUP" = "y" ]; then
echo "Auto-cleanup enabled. Starting cleanup..."
cleanup_resources
echo "Cleanup completed."
elif [ "$AUTO_CLEANUP" = "n" ]; then
echo "Auto-cleanup disabled. Resources will remain in your account."
echo "To clean up later, run this script again with cleanup option 'y'"
else
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
echo "Starting cleanup..."
cleanup_resources
echo "Cleanup completed."
else
echo "Skipping cleanup. Resources will remain in your account."
echo "To clean up later, delete the following resources:"
echo "- Feature Groups: $CUSTOMERS_FEATURE_GROUP_NAME, $ORDERS_FEATURE_GROUP_NAME"
echo "- S3 Bucket: $S3_BUCKET_NAME"
if [[ " ${CREATED_RESOURCES[@]} " =~ " IAMRole:" ]]; then
echo "- IAM Role: $(echo "${CREATED_RESOURCES[@]}" | grep -o 'IAMRole:[^[:space:]]*' | cut -d: -f2)"
fi
echo ""
echo "Estimated ongoing cost: ~$0.01 per month for online store"
fi
fi
echo "Script completed at $(date)"
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AttachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/AttachRolePolicy)
+ [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket)
+ [CreateFeatureGroup](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/CreateFeatureGroup)
+ [CreateRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/CreateRole)
+ [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket)
+ [DeleteFeatureGroup](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/DeleteFeatureGroup)
+ [DeleteRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DeleteRole)
+ [DescribeFeatureGroup](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/DescribeFeatureGroup)
+ [DetachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DetachRolePolicy)
+ [GetCallerIdentity](https://docs.aws.amazon.com/goto/aws-cli/sts-2011-06-15/GetCallerIdentity)
+ [GetRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/GetRole)
+ [ListFeatureGroups](https://docs.aws.amazon.com/goto/aws-cli/sagemaker-2017-07-24/ListFeatureGroups)
+ [PutPublicAccessBlock](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutPublicAccessBlock)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
------
# Getting started with Amazon Textract
The following code example shows how to:
+ Create an S3 bucket
+ Upload a document to S3
+ Clean up resources
------
#### [ Bash ]
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/074-amazon-textract-gs) repository.
```
#!/bin/bash
# Amazon Textract Getting Started Tutorial Script
# This script demonstrates how to use Amazon Textract to analyze document text
# Set up logging
LOG_FILE="textract-tutorial.log"
exec > >(tee -a "$LOG_FILE") 2>&1
echo "==================================================="
echo "Amazon Textract Getting Started Tutorial"
echo "==================================================="
echo "This script will guide you through using Amazon Textract to analyze document text."
echo ""
# Function to check for errors in command output and exit code
check_error() {
local exit_code=$1
local output=$2
local cmd=$3
if [ $exit_code -ne 0 ] || echo "$output" | grep -i "error" > /dev/null; then
echo "ERROR: Command failed: $cmd"
echo "$output"
cleanup_on_error
exit 1
fi
}
# Function to clean up resources on error
cleanup_on_error() {
echo "Error encountered. Cleaning up resources..."
# Clean up temporary JSON files
if [ -f "document.json" ]; then
rm -f document.json
fi
if [ -f "features.json" ]; then
rm -f features.json
fi
if [ -n "$DOCUMENT_NAME" ] && [ -n "$BUCKET_NAME" ]; then
echo "Deleting document from S3..."
aws s3 rm "s3://$BUCKET_NAME/$DOCUMENT_NAME" || echo "Failed to delete document"
fi
if [ -n "$BUCKET_NAME" ]; then
echo "Deleting S3 bucket..."
aws s3 rb "s3://$BUCKET_NAME" --force || echo "Failed to delete bucket"
fi
}
# Verify AWS CLI is installed and configured
echo "Verifying AWS CLI configuration..."
AWS_CONFIG_OUTPUT=$(aws configure list 2>&1)
AWS_CONFIG_STATUS=$?
if [ $AWS_CONFIG_STATUS -ne 0 ]; then
echo "ERROR: AWS CLI is not properly configured."
echo "$AWS_CONFIG_OUTPUT"
exit 1
fi
# Verify AWS region is configured and supports Textract
AWS_REGION=$(aws configure get region)
if [ -z "$AWS_REGION" ]; then
echo "ERROR: No AWS region configured. Please run 'aws configure' to set a default region."
exit 1
fi
# Check if Textract is available in the configured region
echo "Checking if Amazon Textract is available in region $AWS_REGION..."
TEXTRACT_CHECK=$(aws textract help 2>&1)
TEXTRACT_CHECK_STATUS=$?
if [ $TEXTRACT_CHECK_STATUS -ne 0 ]; then
echo "ERROR: Amazon Textract may not be available in region $AWS_REGION."
echo "$TEXTRACT_CHECK"
exit 1
fi
# Generate a random identifier for S3 bucket
RANDOM_ID=$(openssl rand -hex 6)
BUCKET_NAME="textract-${RANDOM_ID}"
DOCUMENT_NAME="document.png"
RESOURCES_CREATED=()
# Step 1: Create S3 bucket
echo "Creating S3 bucket: $BUCKET_NAME"
CREATE_BUCKET_OUTPUT=$(aws s3 mb "s3://$BUCKET_NAME" 2>&1)
CREATE_BUCKET_STATUS=$?
echo "$CREATE_BUCKET_OUTPUT"
check_error $CREATE_BUCKET_STATUS "$CREATE_BUCKET_OUTPUT" "aws s3 mb s3://$BUCKET_NAME"
RESOURCES_CREATED+=("S3 Bucket: $BUCKET_NAME")
# Step 2: Check if sample document exists, if not create a simple one
if [ ! -f "$DOCUMENT_NAME" ]; then
echo "Sample document not found. Please provide a document to analyze."
echo "Enter the path to your document (must be an image file like PNG or JPEG):"
read -r DOCUMENT_PATH
if [ ! -f "$DOCUMENT_PATH" ]; then
echo "File not found: $DOCUMENT_PATH"
cleanup_on_error
exit 1
fi
DOCUMENT_NAME=$(basename "$DOCUMENT_PATH")
echo "Using document: $DOCUMENT_PATH as $DOCUMENT_NAME"
# Copy the document to the current directory
cp "$DOCUMENT_PATH" "./$DOCUMENT_NAME"
fi
# Step 3: Upload document to S3
echo "Uploading document to S3..."
UPLOAD_OUTPUT=$(aws s3 cp "./$DOCUMENT_NAME" "s3://$BUCKET_NAME/" 2>&1)
UPLOAD_STATUS=$?
echo "$UPLOAD_OUTPUT"
check_error $UPLOAD_STATUS "$UPLOAD_OUTPUT" "aws s3 cp ./$DOCUMENT_NAME s3://$BUCKET_NAME/"
RESOURCES_CREATED+=("S3 Object: s3://$BUCKET_NAME/$DOCUMENT_NAME")
# Step 4: Analyze document with Amazon Textract
echo "Analyzing document with Amazon Textract..."
echo "This may take a few seconds..."
# Create a JSON file for the document parameter to avoid shell escaping issues
cat > document.json << EOF
{
"S3Object": {
"Bucket": "$BUCKET_NAME",
"Name": "$DOCUMENT_NAME"
}
}
EOF
# Create a JSON file for the feature types parameter
cat > features.json << EOF
["TABLES","FORMS","SIGNATURES"]
EOF
ANALYZE_OUTPUT=$(aws textract analyze-document --document file://document.json --feature-types file://features.json 2>&1)
ANALYZE_STATUS=$?
echo "Analysis complete."
if [ $ANALYZE_STATUS -ne 0 ]; then
echo "ERROR: Document analysis failed"
echo "$ANALYZE_OUTPUT"
cleanup_on_error
exit 1
fi
# Save the analysis results to a file
echo "$ANALYZE_OUTPUT" > textract-analysis-results.json
echo "Analysis results saved to textract-analysis-results.json"
RESOURCES_CREATED+=("Local file: textract-analysis-results.json")
# Display a summary of the analysis
echo ""
echo "==================================================="
echo "Analysis Summary"
echo "==================================================="
PAGES=$(echo "$ANALYZE_OUTPUT" | grep -o '"Pages": [0-9]*' | awk '{print $2}')
echo "Document pages: $PAGES"
BLOCKS_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType":' | wc -l)
echo "Total blocks detected: $BLOCKS_COUNT"
# Count different block types
PAGE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "PAGE"' | wc -l)
LINE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "LINE"' | wc -l)
WORD_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "WORD"' | wc -l)
TABLE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "TABLE"' | wc -l)
CELL_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "CELL"' | wc -l)
KEY_VALUE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "KEY_VALUE_SET"' | wc -l)
SIGNATURE_COUNT=$(echo "$ANALYZE_OUTPUT" | grep -o '"BlockType": "SIGNATURE"' | wc -l)
echo "Pages: $PAGE_COUNT"
echo "Lines of text: $LINE_COUNT"
echo "Words: $WORD_COUNT"
echo "Tables: $TABLE_COUNT"
echo "Table cells: $CELL_COUNT"
echo "Key-value pairs: $KEY_VALUE_COUNT"
echo "Signatures: $SIGNATURE_COUNT"
echo ""
# Cleanup confirmation
echo ""
echo "==================================================="
echo "RESOURCES CREATED"
echo "==================================================="
for resource in "${RESOURCES_CREATED[@]}"; do
echo "- $resource"
done
echo ""
echo "==================================================="
echo "CLEANUP CONFIRMATION"
echo "==================================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy] ]]; then
echo "Cleaning up resources..."
# Delete document from S3
echo "Deleting document from S3..."
DELETE_DOC_OUTPUT=$(aws s3 rm "s3://$BUCKET_NAME/$DOCUMENT_NAME" 2>&1)
DELETE_DOC_STATUS=$?
echo "$DELETE_DOC_OUTPUT"
check_error $DELETE_DOC_STATUS "$DELETE_DOC_OUTPUT" "aws s3 rm s3://$BUCKET_NAME/$DOCUMENT_NAME"
# Delete S3 bucket
echo "Deleting S3 bucket..."
DELETE_BUCKET_OUTPUT=$(aws s3 rb "s3://$BUCKET_NAME" --force 2>&1)
DELETE_BUCKET_STATUS=$?
echo "$DELETE_BUCKET_OUTPUT"
check_error $DELETE_BUCKET_STATUS "$DELETE_BUCKET_OUTPUT" "aws s3 rb s3://$BUCKET_NAME --force"
# Delete local JSON files
rm -f document.json features.json
echo "Cleanup complete. The analysis results file (textract-analysis-results.json) has been kept."
else
echo "Resources have been preserved."
fi
echo ""
echo "==================================================="
echo "Tutorial complete!"
echo "==================================================="
echo "You have successfully analyzed a document using Amazon Textract."
echo "The analysis results are available in textract-analysis-results.json"
echo ""
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AnalyzeDocument](https://docs.aws.amazon.com/goto/aws-cli/textract-2018-06-27/AnalyzeDocument)
+ [Cp](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Cp)
+ [Help](https://docs.aws.amazon.com/goto/aws-cli/textract-2018-06-27/Help)
+ [Mb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Mb)
+ [Rb](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rb)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
------
# Getting started with Config
The following code example shows how to:
+ Create an Amazon S3 bucket
+ Create an Amazon SNS topic
+ Create an IAM role for Config
+ Set up the Config configuration recorder
+ Set up the Config delivery channel
+ Start the configuration recorder
+ Verify the Config setup
------
#### [ Bash ]
**AWS CLI with Bash script**
There's more on GitHub. Find the complete example and learn how to set up and run in the [Sample developer tutorials](https://github.com/aws-samples/sample-developer-tutorials/tree/main/tuts/053-aws-config-gs) repository.
```
#!/bin/bash
# AWS Config Setup Script (v2)
# This script sets up AWS Config with the AWS CLI
# Error handling
set -e
LOGFILE="aws-config-setup-v2.log"
touch $LOGFILE
exec > >(tee -a $LOGFILE)
exec 2>&1
# Function to handle errors
handle_error() {
echo "ERROR: An error occurred at line $1"
echo "Attempting to clean up resources..."
cleanup_resources
exit 1
}
# Set trap for error handling
trap 'handle_error $LINENO' ERR
# Function to generate random identifier
generate_random_id() {
echo $(openssl rand -hex 6)
}
# Function to check if command was successful
check_command() {
if echo "$1" | grep -i "error" > /dev/null; then
echo "ERROR: $1"
return 1
fi
return 0
}
# Function to clean up resources
cleanup_resources() {
if [ -n "$CONFIG_RECORDER_NAME" ]; then
echo "Stopping configuration recorder..."
aws configservice stop-configuration-recorder --configuration-recorder-name "$CONFIG_RECORDER_NAME" 2>/dev/null || true
fi
# Check if we created a new delivery channel before trying to delete it
if [ -n "$DELIVERY_CHANNEL_NAME" ] && [ "$CREATED_NEW_DELIVERY_CHANNEL" = "true" ]; then
echo "Deleting delivery channel..."
aws configservice delete-delivery-channel --delivery-channel-name "$DELIVERY_CHANNEL_NAME" 2>/dev/null || true
fi
if [ -n "$CONFIG_RECORDER_NAME" ] && [ "$CREATED_NEW_CONFIG_RECORDER" = "true" ]; then
echo "Deleting configuration recorder..."
aws configservice delete-configuration-recorder --configuration-recorder-name "$CONFIG_RECORDER_NAME" 2>/dev/null || true
fi
if [ -n "$ROLE_NAME" ]; then
if [ -n "$POLICY_NAME" ]; then
echo "Detaching custom policy from role..."
aws iam delete-role-policy --role-name "$ROLE_NAME" --policy-name "$POLICY_NAME" 2>/dev/null || true
fi
if [ -n "$MANAGED_POLICY_ARN" ]; then
echo "Detaching managed policy from role..."
aws iam detach-role-policy --role-name "$ROLE_NAME" --policy-arn "$MANAGED_POLICY_ARN" 2>/dev/null || true
fi
echo "Deleting IAM role..."
aws iam delete-role --role-name "$ROLE_NAME" 2>/dev/null || true
fi
if [ -n "$SNS_TOPIC_ARN" ]; then
echo "Deleting SNS topic..."
aws sns delete-topic --topic-arn "$SNS_TOPIC_ARN" 2>/dev/null || true
fi
if [ -n "$S3_BUCKET_NAME" ]; then
echo "Emptying S3 bucket..."
aws s3 rm "s3://$S3_BUCKET_NAME" --recursive 2>/dev/null || true
echo "Deleting S3 bucket..."
aws s3api delete-bucket --bucket "$S3_BUCKET_NAME" 2>/dev/null || true
fi
}
# Function to display created resources
display_resources() {
echo ""
echo "==========================================="
echo "CREATED RESOURCES"
echo "==========================================="
echo "S3 Bucket: $S3_BUCKET_NAME"
echo "SNS Topic ARN: $SNS_TOPIC_ARN"
echo "IAM Role: $ROLE_NAME"
if [ "$CREATED_NEW_CONFIG_RECORDER" = "true" ]; then
echo "Configuration Recorder: $CONFIG_RECORDER_NAME (newly created)"
else
echo "Configuration Recorder: $CONFIG_RECORDER_NAME (existing)"
fi
if [ "$CREATED_NEW_DELIVERY_CHANNEL" = "true" ]; then
echo "Delivery Channel: $DELIVERY_CHANNEL_NAME (newly created)"
else
echo "Delivery Channel: $DELIVERY_CHANNEL_NAME (existing)"
fi
echo "==========================================="
}
# Get AWS account ID
echo "Getting AWS account ID..."
ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
if [ -z "$ACCOUNT_ID" ]; then
echo "ERROR: Failed to get AWS account ID"
exit 1
fi
echo "AWS Account ID: $ACCOUNT_ID"
# Generate random identifier for resources
RANDOM_ID=$(generate_random_id)
echo "Generated random identifier: $RANDOM_ID"
# Step 1: Create an S3 bucket
S3_BUCKET_NAME="configservice-${RANDOM_ID}"
echo "Creating S3 bucket: $S3_BUCKET_NAME"
# Get the current region
AWS_REGION=$(aws configure get region)
if [ -z "$AWS_REGION" ]; then
AWS_REGION="us-east-1" # Default to us-east-1 if no region is configured
fi
echo "Using AWS Region: $AWS_REGION"
# Create bucket with appropriate command based on region
if [ "$AWS_REGION" = "us-east-1" ]; then
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME")
else
BUCKET_RESULT=$(aws s3api create-bucket --bucket "$S3_BUCKET_NAME" --create-bucket-configuration LocationConstraint="$AWS_REGION")
fi
check_command "$BUCKET_RESULT"
echo "S3 bucket created: $S3_BUCKET_NAME"
# Block public access for the bucket
aws s3api put-public-access-block \
--bucket "$S3_BUCKET_NAME" \
--public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
echo "Public access blocked for bucket"
# Step 2: Create an SNS topic
TOPIC_NAME="config-topic-${RANDOM_ID}"
echo "Creating SNS topic: $TOPIC_NAME"
SNS_RESULT=$(aws sns create-topic --name "$TOPIC_NAME")
check_command "$SNS_RESULT"
SNS_TOPIC_ARN=$(echo "$SNS_RESULT" | grep -o 'arn:aws:sns:[^"]*')
echo "SNS topic created: $SNS_TOPIC_ARN"
# Step 3: Create an IAM role for AWS Config
ROLE_NAME="config-role-${RANDOM_ID}"
POLICY_NAME="config-delivery-permissions"
MANAGED_POLICY_ARN="arn:aws:iam::aws:policy/service-role/AWS_ConfigRole"
echo "Creating trust policy document..."
cat > config-trust-policy.json << EOF
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "config.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EOF
echo "Creating IAM role: $ROLE_NAME"
ROLE_RESULT=$(aws iam create-role --role-name "$ROLE_NAME" --assume-role-policy-document file://config-trust-policy.json)
check_command "$ROLE_RESULT"
ROLE_ARN=$(echo "$ROLE_RESULT" | grep -o 'arn:aws:iam::[^"]*' | head -1)
echo "IAM role created: $ROLE_ARN"
echo "Attaching AWS managed policy to role..."
ATTACH_RESULT=$(aws iam attach-role-policy --role-name "$ROLE_NAME" --policy-arn "$MANAGED_POLICY_ARN")
check_command "$ATTACH_RESULT"
echo "AWS managed policy attached"
echo "Creating custom policy document for S3 and SNS access..."
cat > config-delivery-permissions.json << EOF
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::${S3_BUCKET_NAME}/AWSLogs/${ACCOUNT_ID}/*",
"Condition": {
"StringLike": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetBucketAcl"
],
"Resource": "arn:aws:s3:::${S3_BUCKET_NAME}"
},
{
"Effect": "Allow",
"Action": [
"sns:Publish"
],
"Resource": "${SNS_TOPIC_ARN}"
}
]
}
EOF
echo "Attaching custom policy to role..."
POLICY_RESULT=$(aws iam put-role-policy --role-name "$ROLE_NAME" --policy-name "$POLICY_NAME" --policy-document file://config-delivery-permissions.json)
check_command "$POLICY_RESULT"
echo "Custom policy attached"
# Wait for IAM role to propagate
echo "Waiting for IAM role to propagate (15 seconds)..."
sleep 15
# Step 4: Check if configuration recorder already exists
CONFIG_RECORDER_NAME="default"
CREATED_NEW_CONFIG_RECORDER="false"
echo "Checking for existing configuration recorder..."
EXISTING_RECORDERS=$(aws configservice describe-configuration-recorders 2>/dev/null || echo "")
if echo "$EXISTING_RECORDERS" | grep -q "name"; then
echo "Configuration recorder already exists. Will update it."
# Get the name of the existing recorder
CONFIG_RECORDER_NAME=$(echo "$EXISTING_RECORDERS" | grep -o '"name": "[^"]*"' | head -1 | cut -d'"' -f4)
echo "Using existing configuration recorder: $CONFIG_RECORDER_NAME"
else
echo "No existing configuration recorder found. Will create a new one."
CREATED_NEW_CONFIG_RECORDER="true"
fi
echo "Creating configuration recorder configuration..."
cat > configurationRecorder.json << EOF
{
"name": "${CONFIG_RECORDER_NAME}",
"roleARN": "${ROLE_ARN}",
"recordingMode": {
"recordingFrequency": "CONTINUOUS"
}
}
EOF
echo "Creating recording group configuration..."
cat > recordingGroup.json << EOF
{
"allSupported": true,
"includeGlobalResourceTypes": true
}
EOF
echo "Setting up configuration recorder..."
RECORDER_RESULT=$(aws configservice put-configuration-recorder --configuration-recorder file://configurationRecorder.json --recording-group file://recordingGroup.json)
check_command "$RECORDER_RESULT"
echo "Configuration recorder set up"
# Step 5: Check if delivery channel already exists
DELIVERY_CHANNEL_NAME="default"
CREATED_NEW_DELIVERY_CHANNEL="false"
echo "Checking for existing delivery channel..."
EXISTING_CHANNELS=$(aws configservice describe-delivery-channels 2>/dev/null || echo "")
if echo "$EXISTING_CHANNELS" | grep -q "name"; then
echo "Delivery channel already exists."
# Get the name of the existing channel
DELIVERY_CHANNEL_NAME=$(echo "$EXISTING_CHANNELS" | grep -o '"name": "[^"]*"' | head -1 | cut -d'"' -f4)
echo "Using existing delivery channel: $DELIVERY_CHANNEL_NAME"
# Update the existing delivery channel
echo "Creating delivery channel configuration for update..."
cat > deliveryChannel.json << EOF
{
"name": "${DELIVERY_CHANNEL_NAME}",
"s3BucketName": "${S3_BUCKET_NAME}",
"snsTopicARN": "${SNS_TOPIC_ARN}",
"configSnapshotDeliveryProperties": {
"deliveryFrequency": "Six_Hours"
}
}
EOF
echo "Updating delivery channel..."
CHANNEL_RESULT=$(aws configservice put-delivery-channel --delivery-channel file://deliveryChannel.json)
check_command "$CHANNEL_RESULT"
echo "Delivery channel updated"
else
echo "No existing delivery channel found. Will create a new one."
CREATED_NEW_DELIVERY_CHANNEL="true"
echo "Creating delivery channel configuration..."
cat > deliveryChannel.json << EOF
{
"name": "${DELIVERY_CHANNEL_NAME}",
"s3BucketName": "${S3_BUCKET_NAME}",
"snsTopicARN": "${SNS_TOPIC_ARN}",
"configSnapshotDeliveryProperties": {
"deliveryFrequency": "Six_Hours"
}
}
EOF
echo "Creating delivery channel..."
CHANNEL_RESULT=$(aws configservice put-delivery-channel --delivery-channel file://deliveryChannel.json)
check_command "$CHANNEL_RESULT"
echo "Delivery channel created"
fi
# Step 6: Start the configuration recorder
echo "Checking configuration recorder status..."
RECORDER_STATUS=$(aws configservice describe-configuration-recorder-status 2>/dev/null || echo "")
if echo "$RECORDER_STATUS" | grep -q '"recording": true'; then
echo "Configuration recorder is already running."
else
echo "Starting configuration recorder..."
START_RESULT=$(aws configservice start-configuration-recorder --configuration-recorder-name "$CONFIG_RECORDER_NAME")
check_command "$START_RESULT"
echo "Configuration recorder started"
fi
# Step 7: Verify the AWS Config setup
echo "Verifying delivery channel..."
VERIFY_CHANNEL=$(aws configservice describe-delivery-channels)
check_command "$VERIFY_CHANNEL"
echo "$VERIFY_CHANNEL"
echo "Verifying configuration recorder..."
VERIFY_RECORDER=$(aws configservice describe-configuration-recorders)
check_command "$VERIFY_RECORDER"
echo "$VERIFY_RECORDER"
echo "Verifying configuration recorder status..."
VERIFY_STATUS=$(aws configservice describe-configuration-recorder-status)
check_command "$VERIFY_STATUS"
echo "$VERIFY_STATUS"
# Display created resources
display_resources
# Ask if user wants to clean up resources
echo ""
echo "==========================================="
echo "CLEANUP CONFIRMATION"
echo "==========================================="
echo "Do you want to clean up all created resources? (y/n): "
read -r CLEANUP_CHOICE
if [[ "$CLEANUP_CHOICE" =~ ^[Yy]$ ]]; then
echo "Cleaning up resources..."
cleanup_resources
echo "Cleanup completed."
else
echo "Resources will not be cleaned up. You can manually clean them up later."
fi
echo "Script completed successfully!"
```
+ For API details, see the following topics in *AWS CLI Command Reference*.
+ [AttachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/AttachRolePolicy)
+ [CreateBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/CreateBucket)
+ [CreateRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/CreateRole)
+ [CreateTopic](https://docs.aws.amazon.com/goto/aws-cli/sns-2010-03-31/CreateTopic)
+ [DeleteBucket](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/DeleteBucket)
+ [DeleteConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DeleteConfigurationRecorder)
+ [DeleteDeliveryChannel](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DeleteDeliveryChannel)
+ [DeleteRole](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DeleteRole)
+ [DeleteRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DeleteRolePolicy)
+ [DeleteTopic](https://docs.aws.amazon.com/goto/aws-cli/sns-2010-03-31/DeleteTopic)
+ [DescribeConfigurationRecorderStatus](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DescribeConfigurationRecorderStatus)
+ [DescribeConfigurationRecorders](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DescribeConfigurationRecorders)
+ [DescribeDeliveryChannels](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/DescribeDeliveryChannels)
+ [DetachRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/DetachRolePolicy)
+ [GetCallerIdentity](https://docs.aws.amazon.com/goto/aws-cli/sts-2011-06-15/GetCallerIdentity)
+ [PutConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/PutConfigurationRecorder)
+ [PutDeliveryChannel](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/PutDeliveryChannel)
+ [PutPublicAccessBlock](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/PutPublicAccessBlock)
+ [PutRolePolicy](https://docs.aws.amazon.com/goto/aws-cli/iam-2010-05-08/PutRolePolicy)
+ [Rm](https://docs.aws.amazon.com/goto/aws-cli/s3-2006-03-01/Rm)
+ [StartConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/StartConfigurationRecorder)
+ [StopConfigurationRecorder](https://docs.aws.amazon.com/goto/aws-cli/config-2014-11-12/StopConfigurationRecorder)
------
# Work with Amazon S3 object lock features using an AWS SDK
The following code examples show how to work with S3 object lock features.
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/scenarios/S3ObjectLockScenario#code-examples).
Run an interactive scenario demonstrating Amazon S3 object lock features.
```
using Amazon.S3;
using Amazon.S3.Model;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Console;
using Microsoft.Extensions.Logging.Debug;
namespace S3ObjectLockScenario;
public static class S3ObjectLockWorkflow
{
/*
Before running this .NET code example, set up your development environment, including your credentials.
This .NET example performs the following tasks:
1. Create test Amazon Simple Storage Service (S3) buckets with different lock policies.
2. Upload sample objects to each bucket.
3. Set some Legal Hold and Retention Periods on objects and buckets.
4. Investigate lock policies by viewing settings or attempting to delete or overwrite objects.
5. Clean up objects and buckets.
*/
public static S3ActionsWrapper _s3ActionsWrapper = null!;
public static IConfiguration _configuration = null!;
private static string _resourcePrefix = null!;
private static string noLockBucketName = null!;
private static string lockEnabledBucketName = null!;
private static string retentionAfterCreationBucketName = null!;
private static List bucketNames = new List();
private static List fileNames = new List();
public static async Task Main(string[] args)
{
// Set up dependency injection for the Amazon service.
using var host = Host.CreateDefaultBuilder(args)
.ConfigureLogging(logging =>
logging.AddFilter("System", LogLevel.Debug)
.AddFilter("Microsoft", LogLevel.Information)
.AddFilter("Microsoft", LogLevel.Trace))
.ConfigureServices((_, services) =>
services.AddAWSService()
.AddTransient()
)
.Build();
_configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("settings.json") // Load settings from .json file.
.AddJsonFile("settings.local.json",
true) // Optionally, load local settings.
.Build();
ConfigurationSetup();
ServicesSetup(host);
try
{
Console.WriteLine(new string('-', 80));
Console.WriteLine("Welcome to the Amazon Simple Storage Service (S3) Object Locking Feature Scenario.");
Console.WriteLine(new string('-', 80));
await Setup(true);
await DemoActionChoices();
Console.WriteLine(new string('-', 80));
Console.WriteLine("Cleaning up resources.");
Console.WriteLine(new string('-', 80));
await Cleanup(true);
Console.WriteLine(new string('-', 80));
Console.WriteLine("Amazon S3 Object Locking Scenario is complete.");
Console.WriteLine(new string('-', 80));
}
catch (Exception ex)
{
Console.WriteLine(new string('-', 80));
Console.WriteLine($"There was a problem: {ex.Message}");
await Cleanup(true);
Console.WriteLine(new string('-', 80));
}
}
///
/// Populate the services for use within the console application.
///
/// The services host.
private static void ServicesSetup(IHost host)
{
_s3ActionsWrapper = host.Services.GetRequiredService();
}
///
/// Any setup operations needed.
///
public static void ConfigurationSetup()
{
_resourcePrefix = _configuration["resourcePrefix"] ?? "dotnet-example";
noLockBucketName = _resourcePrefix + "-no-lock";
lockEnabledBucketName = _resourcePrefix + "-lock-enabled";
retentionAfterCreationBucketName = _resourcePrefix + "-retention-after-creation";
bucketNames.Add(noLockBucketName);
bucketNames.Add(lockEnabledBucketName);
bucketNames.Add(retentionAfterCreationBucketName);
}
//
/// Deploy necessary resources for the scenario.
///
/// True to run as interactive.
/// True if successful.
public static async Task Setup(bool interactive)
{
Console.WriteLine(
"\nFor this scenario, we will use the AWS SDK for .NET to create several S3\n" +
"buckets and files to demonstrate working with S3 locking features.\n");
Console.WriteLine(new string('-', 80));
Console.WriteLine("Press Enter when you are ready to start.");
if (interactive)
Console.ReadLine();
Console.WriteLine("\nS3 buckets can be created either with or without object lock enabled.");
await _s3ActionsWrapper.CreateBucketWithObjectLock(noLockBucketName, false);
await _s3ActionsWrapper.CreateBucketWithObjectLock(lockEnabledBucketName, true);
await _s3ActionsWrapper.CreateBucketWithObjectLock(retentionAfterCreationBucketName, false);
Console.WriteLine("Press Enter to continue.");
if (interactive)
Console.ReadLine();
Console.WriteLine("\nA bucket can be configured to use object locking with a default retention period.");
await _s3ActionsWrapper.ModifyBucketDefaultRetention(retentionAfterCreationBucketName, true,
ObjectLockRetentionMode.Governance, DateTime.UtcNow.AddDays(1));
Console.WriteLine("Press Enter to continue.");
if (interactive)
Console.ReadLine();
Console.WriteLine("\nObject lock policies can also be added to existing buckets.");
await _s3ActionsWrapper.EnableObjectLockOnBucket(lockEnabledBucketName);
Console.WriteLine("Press Enter to continue.");
if (interactive)
Console.ReadLine();
// Upload some files to the buckets.
Console.WriteLine("\nNow let's add some test files:");
var fileName = _configuration["exampleFileName"] ?? "exampleFile.txt";
int fileCount = 2;
// Create the file if it does not already exist.
if (!File.Exists(fileName))
{
await using StreamWriter sw = File.CreateText(fileName);
await sw.WriteLineAsync(
"This is a sample file for uploading to a bucket.");
}
foreach (var bucketName in bucketNames)
{
for (int i = 0; i < fileCount; i++)
{
var numberedFileName = Path.GetFileNameWithoutExtension(fileName) + i + Path.GetExtension(fileName);
fileNames.Add(numberedFileName);
await _s3ActionsWrapper.UploadFileAsync(bucketName, numberedFileName, fileName);
}
}
Console.WriteLine("Press Enter to continue.");
if (interactive)
Console.ReadLine();
if (!interactive)
return true;
Console.WriteLine("\nNow we can set some object lock policies on individual files:");
foreach (var bucketName in bucketNames)
{
for (int i = 0; i < fileNames.Count; i++)
{
// No modifications to the objects in the first bucket.
if (bucketName != bucketNames[0])
{
var exampleFileName = fileNames[i];
switch (i)
{
case 0:
{
var question =
$"\nWould you like to add a legal hold to {exampleFileName} in {bucketName}? (y/n)";
if (GetYesNoResponse(question))
{
// Set a legal hold.
await _s3ActionsWrapper.ModifyObjectLegalHold(bucketName, exampleFileName, ObjectLockLegalHoldStatus.On);
}
break;
}
case 1:
{
var question =
$"\nWould you like to add a 1 day Governance retention period to {exampleFileName} in {bucketName}? (y/n)" +
"\nReminder: Only a user with the s3:BypassGovernanceRetention permission will be able to delete this file or its bucket until the retention period has expired.";
if (GetYesNoResponse(question))
{
// Set a Governance mode retention period for 1 day.
await _s3ActionsWrapper.ModifyObjectRetentionPeriod(
bucketName, exampleFileName,
ObjectLockRetentionMode.Governance,
DateTime.UtcNow.AddDays(1));
}
break;
}
}
}
}
}
Console.WriteLine(new string('-', 80));
return true;
}
//
/// List all of the current buckets and objects.
///
/// True to run as interactive.
/// The list of buckets and objects.
public static async Task> ListBucketsAndObjects(bool interactive)
{
var allObjects = new List();
foreach (var bucketName in bucketNames)
{
var objectsInBucket = await _s3ActionsWrapper.ListBucketObjectsAndVersions(bucketName);
foreach (var objectKey in objectsInBucket.Versions)
{
allObjects.Add(objectKey);
}
}
if (interactive)
{
Console.WriteLine("\nCurrent buckets and objects:\n");
int i = 0;
foreach (var bucketObject in allObjects)
{
i++;
Console.WriteLine(
$"{i}: {bucketObject.Key} \n\tBucket: {bucketObject.BucketName}\n\tVersion: {bucketObject.VersionId}");
}
}
return allObjects;
}
///
/// Present the user with the demo action choices.
///
/// Async task.
public static async Task DemoActionChoices()
{
var choices = new string[]{
"List all files in buckets.",
"Attempt to delete a file.",
"Attempt to delete a file with retention period bypass.",
"Attempt to overwrite a file.",
"View the object and bucket retention settings for a file.",
"View the legal hold settings for a file.",
"Finish the scenario."};
var choice = 0;
// Keep asking the user until they choose to move on.
while (choice != 6)
{
Console.WriteLine(new string('-', 80));
choice = GetChoiceResponse(
"\nExplore the S3 locking features by selecting one of the following choices:"
, choices);
Console.WriteLine(new string('-', 80));
switch (choice)
{
case 0:
{
await ListBucketsAndObjects(true);
break;
}
case 1:
{
Console.WriteLine("\nEnter the number of the object to delete:");
var allFiles = await ListBucketsAndObjects(true);
var fileChoice = GetChoiceResponse(null, allFiles.Select(f => f.Key).ToArray());
await _s3ActionsWrapper.DeleteObjectFromBucket(allFiles[fileChoice].BucketName, allFiles[fileChoice].Key, false, allFiles[fileChoice].VersionId);
break;
}
case 2:
{
Console.WriteLine("\nEnter the number of the object to delete:");
var allFiles = await ListBucketsAndObjects(true);
var fileChoice = GetChoiceResponse(null, allFiles.Select(f => f.Key).ToArray());
await _s3ActionsWrapper.DeleteObjectFromBucket(allFiles[fileChoice].BucketName, allFiles[fileChoice].Key, true, allFiles[fileChoice].VersionId);
break;
}
case 3:
{
var allFiles = await ListBucketsAndObjects(true);
Console.WriteLine("\nEnter the number of the object to overwrite:");
var fileChoice = GetChoiceResponse(null, allFiles.Select(f => f.Key).ToArray());
// Create the file if it does not already exist.
if (!File.Exists(allFiles[fileChoice].Key))
{
await using StreamWriter sw = File.CreateText(allFiles[fileChoice].Key);
await sw.WriteLineAsync(
"This is a sample file for uploading to a bucket.");
}
await _s3ActionsWrapper.UploadFileAsync(allFiles[fileChoice].BucketName, allFiles[fileChoice].Key, allFiles[fileChoice].Key);
break;
}
case 4:
{
var allFiles = await ListBucketsAndObjects(true);
Console.WriteLine("\nEnter the number of the object and bucket to view:");
var fileChoice = GetChoiceResponse(null, allFiles.Select(f => f.Key).ToArray());
await _s3ActionsWrapper.GetObjectRetention(allFiles[fileChoice].BucketName, allFiles[fileChoice].Key);
await _s3ActionsWrapper.GetBucketObjectLockConfiguration(allFiles[fileChoice].BucketName);
break;
}
case 5:
{
var allFiles = await ListBucketsAndObjects(true);
Console.WriteLine("\nEnter the number of the object to view:");
var fileChoice = GetChoiceResponse(null, allFiles.Select(f => f.Key).ToArray());
await _s3ActionsWrapper.GetObjectLegalHold(allFiles[fileChoice].BucketName, allFiles[fileChoice].Key);
break;
}
}
}
return true;
}
//
/// Clean up the resources from the scenario.
///
/// True to run as interactive.
/// True if successful.
public static async Task Cleanup(bool interactive)
{
Console.WriteLine(new string('-', 80));
if (!interactive || GetYesNoResponse("Do you want to clean up all files and buckets? (y/n) "))
{
// Remove all locks and delete all buckets and objects.
var allFiles = await ListBucketsAndObjects(false);
foreach (var fileInfo in allFiles)
{
// Check for a legal hold.
var legalHold = await _s3ActionsWrapper.GetObjectLegalHold(fileInfo.BucketName, fileInfo.Key);
if (legalHold?.Status?.Value == ObjectLockLegalHoldStatus.On)
{
await _s3ActionsWrapper.ModifyObjectLegalHold(fileInfo.BucketName, fileInfo.Key, ObjectLockLegalHoldStatus.Off);
}
// Check for a retention period.
var retention = await _s3ActionsWrapper.GetObjectRetention(fileInfo.BucketName, fileInfo.Key);
var hasRetentionPeriod = retention?.Mode == ObjectLockRetentionMode.Governance && retention.RetainUntilDate > DateTime.UtcNow.Date;
await _s3ActionsWrapper.DeleteObjectFromBucket(fileInfo.BucketName, fileInfo.Key, hasRetentionPeriod, fileInfo.VersionId);
}
foreach (var bucketName in bucketNames)
{
await _s3ActionsWrapper.DeleteBucketByName(bucketName);
}
}
else
{
Console.WriteLine(
"Ok, we'll leave the resources intact.\n" +
"Don't forget to delete them when you're done with them or you might incur unexpected charges."
);
}
Console.WriteLine(new string('-', 80));
return true;
}
///
/// Helper method to get a yes or no response from the user.
///
/// The question string to print on the console.
/// True if the user responds with a yes.
private static bool GetYesNoResponse(string question)
{
Console.WriteLine(question);
var ynResponse = Console.ReadLine();
var response = ynResponse != null && ynResponse.Equals("y", StringComparison.InvariantCultureIgnoreCase);
return response;
}
///
/// Helper method to get a choice response from the user.
///
/// The question string to print on the console.
/// The choices to print on the console.
/// The index of the selected choice
private static int GetChoiceResponse(string? question, string[] choices)
{
if (question != null)
{
Console.WriteLine(question);
for (int i = 0; i < choices.Length; i++)
{
Console.WriteLine($"\t{i + 1}. {choices[i]}");
}
}
var choiceNumber = 0;
while (choiceNumber < 1 || choiceNumber > choices.Length)
{
var choice = Console.ReadLine();
Int32.TryParse(choice, out choiceNumber);
}
return choiceNumber - 1;
}
}
```
A wrapper class for S3 functions.
```
using System.Net;
using Amazon.S3;
using Amazon.S3.Model;
using Microsoft.Extensions.Configuration;
namespace S3ObjectLockScenario;
///
/// Encapsulate the Amazon S3 operations.
///
public class S3ActionsWrapper
{
private readonly IAmazonS3 _amazonS3;
///
/// Constructor for the S3ActionsWrapper.
///
/// The injected S3 client.
public S3ActionsWrapper(IAmazonS3 amazonS3, IConfiguration configuration)
{
_amazonS3 = amazonS3;
}
///
/// Create a new Amazon S3 bucket with object lock actions.
///
/// The name of the bucket to create.
/// True to enable object lock on the bucket.
/// True if successful.
public async Task CreateBucketWithObjectLock(string bucketName, bool enableObjectLock)
{
Console.WriteLine($"\tCreating bucket {bucketName} with object lock {enableObjectLock}.");
try
{
var request = new PutBucketRequest
{
BucketName = bucketName,
UseClientRegion = true,
ObjectLockEnabledForBucket = enableObjectLock,
};
var response = await _amazonS3.PutBucketAsync(request);
return response.HttpStatusCode == System.Net.HttpStatusCode.OK;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"Error creating bucket: '{ex.Message}'");
return false;
}
}
///
/// Enable object lock on an existing bucket.
///
/// The name of the bucket to modify.
/// True if successful.
public async Task EnableObjectLockOnBucket(string bucketName)
{
try
{
// First, enable Versioning on the bucket.
await _amazonS3.PutBucketVersioningAsync(new PutBucketVersioningRequest()
{
BucketName = bucketName,
VersioningConfig = new S3BucketVersioningConfig()
{
EnableMfaDelete = false,
Status = VersionStatus.Enabled
}
});
var request = new PutObjectLockConfigurationRequest()
{
BucketName = bucketName,
ObjectLockConfiguration = new ObjectLockConfiguration()
{
ObjectLockEnabled = new ObjectLockEnabled("Enabled"),
},
};
var response = await _amazonS3.PutObjectLockConfigurationAsync(request);
Console.WriteLine($"\tAdded an object lock policy to bucket {bucketName}.");
return response.HttpStatusCode == System.Net.HttpStatusCode.OK;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"Error modifying object lock: '{ex.Message}'");
return false;
}
}
///
/// Set or modify a retention period on an object in an S3 bucket.
///
/// The bucket of the object.
/// The key of the object.
/// The retention mode.
/// The date retention expires.
/// True if successful.
public async Task ModifyObjectRetentionPeriod(string bucketName,
string objectKey, ObjectLockRetentionMode retention, DateTime retainUntilDate)
{
try
{
var request = new PutObjectRetentionRequest()
{
BucketName = bucketName,
Key = objectKey,
Retention = new ObjectLockRetention()
{
Mode = retention,
RetainUntilDate = retainUntilDate
}
};
var response = await _amazonS3.PutObjectRetentionAsync(request);
Console.WriteLine($"\tSet retention for {objectKey} in {bucketName} until {retainUntilDate:d}.");
return response.HttpStatusCode == System.Net.HttpStatusCode.OK;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tError modifying retention period: '{ex.Message}'");
return false;
}
}
///
/// Set or modify a retention period on an S3 bucket.
///
/// The bucket to modify.
/// The retention mode.
/// The date for retention until.
/// True if successful.
public async Task ModifyBucketDefaultRetention(string bucketName, bool enableObjectLock, ObjectLockRetentionMode retention, DateTime retainUntilDate)
{
var enabledString = enableObjectLock ? "Enabled" : "Disabled";
var timeDifference = retainUntilDate.Subtract(DateTime.Now);
try
{
// First, enable Versioning on the bucket.
await _amazonS3.PutBucketVersioningAsync(new PutBucketVersioningRequest()
{
BucketName = bucketName,
VersioningConfig = new S3BucketVersioningConfig()
{
EnableMfaDelete = false,
Status = VersionStatus.Enabled
}
});
var request = new PutObjectLockConfigurationRequest()
{
BucketName = bucketName,
ObjectLockConfiguration = new ObjectLockConfiguration()
{
ObjectLockEnabled = new ObjectLockEnabled(enabledString),
Rule = new ObjectLockRule()
{
DefaultRetention = new DefaultRetention()
{
Mode = retention,
Days = timeDifference.Days // Can be specified in days or years but not both.
}
}
}
};
var response = await _amazonS3.PutObjectLockConfigurationAsync(request);
Console.WriteLine($"\tAdded a default retention to bucket {bucketName}.");
return response.HttpStatusCode == System.Net.HttpStatusCode.OK;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tError modifying object lock: '{ex.Message}'");
return false;
}
}
///
/// Get the retention period for an S3 object.
///
/// The bucket of the object.
/// The object key.
/// The object retention details.
public async Task GetObjectRetention(string bucketName,
string objectKey)
{
try
{
var request = new GetObjectRetentionRequest()
{
BucketName = bucketName,
Key = objectKey
};
var response = await _amazonS3.GetObjectRetentionAsync(request);
Console.WriteLine($"\tObject retention for {objectKey} in {bucketName}: " +
$"\n\t{response.Retention.Mode} until {response.Retention.RetainUntilDate:d}.");
return response.Retention;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tUnable to fetch object lock retention: '{ex.Message}'");
return new ObjectLockRetention();
}
}
///
/// Set or modify a legal hold on an object in an S3 bucket.
///
/// The bucket of the object.
/// The key of the object.
/// The On or Off status for the legal hold.
/// True if successful.
public async Task ModifyObjectLegalHold(string bucketName,
string objectKey, ObjectLockLegalHoldStatus holdStatus)
{
try
{
var request = new PutObjectLegalHoldRequest()
{
BucketName = bucketName,
Key = objectKey,
LegalHold = new ObjectLockLegalHold()
{
Status = holdStatus
}
};
var response = await _amazonS3.PutObjectLegalHoldAsync(request);
Console.WriteLine($"\tModified legal hold for {objectKey} in {bucketName}.");
return response.HttpStatusCode == System.Net.HttpStatusCode.OK;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tError modifying legal hold: '{ex.Message}'");
return false;
}
}
///
/// Get the legal hold details for an S3 object.
///
/// The bucket of the object.
/// The object key.
/// The object legal hold details.
public async Task GetObjectLegalHold(string bucketName,
string objectKey)
{
try
{
var request = new GetObjectLegalHoldRequest()
{
BucketName = bucketName,
Key = objectKey
};
var response = await _amazonS3.GetObjectLegalHoldAsync(request);
Console.WriteLine($"\tObject legal hold for {objectKey} in {bucketName}: " +
$"\n\tStatus: {response.LegalHold.Status}");
return response.LegalHold;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tUnable to fetch legal hold: '{ex.Message}'");
return new ObjectLockLegalHold();
}
}
///
/// Get the object lock configuration details for an S3 bucket.
///
/// The bucket to get details.
/// The bucket's object lock configuration details.
public async Task GetBucketObjectLockConfiguration(string bucketName)
{
try
{
var request = new GetObjectLockConfigurationRequest()
{
BucketName = bucketName
};
var response = await _amazonS3.GetObjectLockConfigurationAsync(request);
Console.WriteLine($"\tBucket object lock config for {bucketName} in {bucketName}: " +
$"\n\tEnabled: {response.ObjectLockConfiguration.ObjectLockEnabled}" +
$"\n\tRule: {response.ObjectLockConfiguration.Rule?.DefaultRetention}");
return response.ObjectLockConfiguration;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tUnable to fetch object lock config: '{ex.Message}'");
return new ObjectLockConfiguration();
}
}
///
/// Upload a file from the local computer to an Amazon S3 bucket.
///
/// The Amazon S3 bucket to use.
/// The object to upload.
/// The path, including file name, of the object to upload.
/// True if success.
public async Task UploadFileAsync(string bucketName, string objectName, string filePath)
{
var request = new PutObjectRequest
{
BucketName = bucketName,
Key = objectName,
FilePath = filePath,
ChecksumAlgorithm = ChecksumAlgorithm.SHA256
};
var response = await _amazonS3.PutObjectAsync(request);
if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
{
Console.WriteLine($"\tSuccessfully uploaded {objectName} to {bucketName}.");
return true;
}
else
{
Console.WriteLine($"\tCould not upload {objectName} to {bucketName}.");
return false;
}
}
///
/// List bucket objects and versions.
///
/// The Amazon S3 bucket to use.
/// The list of objects and versions.
public async Task ListBucketObjectsAndVersions(string bucketName)
{
var request = new ListVersionsRequest()
{
BucketName = bucketName
};
var response = await _amazonS3.ListVersionsAsync(request);
return response;
}
///
/// Delete an object from a specific bucket.
///
/// The Amazon S3 bucket to use.
/// The key of the object to delete.
/// True if the object has retention settings.
/// Optional versionId.
/// True if successful.
public async Task DeleteObjectFromBucket(string bucketName, string objectKey, bool hasRetention, string? versionId = null)
{
try
{
var request = new DeleteObjectRequest()
{
BucketName = bucketName,
Key = objectKey,
VersionId = versionId,
};
if (hasRetention)
{
// Set the BypassGovernanceRetention header
// if the file has retention settings.
request.BypassGovernanceRetention = true;
}
await _amazonS3.DeleteObjectAsync(request);
Console.WriteLine(
$"Deleted {objectKey} in {bucketName}.");
return true;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tUnable to delete object {objectKey} in bucket {bucketName}: " + ex.Message);
return false;
}
}
///
/// Delete a specific bucket.
///
/// The Amazon S3 bucket to use.
/// The key of the object to delete.
/// Optional versionId.
/// True if successful.
public async Task DeleteBucketByName(string bucketName)
{
try
{
var request = new DeleteBucketRequest() { BucketName = bucketName, };
var response = await _amazonS3.DeleteBucketAsync(request);
Console.WriteLine($"\tDelete for {bucketName} complete.");
return response.HttpStatusCode == HttpStatusCode.OK;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tUnable to delete bucket {bucketName}: " + ex.Message);
return false;
}
}
}
```
+ For API details, see the following topics in *AWS SDK for .NET API Reference*.
+ [GetObjectLegalHold](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObjectLegalHold)
+ [GetObjectLockConfiguration](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObjectLockConfiguration)
+ [GetObjectRetention](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObjectRetention)
+ [PutObjectLegalHold](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/PutObjectLegalHold)
+ [PutObjectLockConfiguration](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/PutObjectLockConfiguration)
+ [PutObjectRetention](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/PutObjectRetention)
------
#### [ Go ]
**SDK for Go V2**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/gov2/workflows/s3_object_lock#code-examples).
Run an interactive scenario demonstrating Amazon S3 object lock features.
```
import (
"context"
"fmt"
"log"
"strings"
"s3_object_lock/actions"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/awsdocs/aws-doc-sdk-examples/gov2/demotools"
)
// ObjectLockScenario contains the steps to run the S3 Object Lock workflow.
type ObjectLockScenario struct {
questioner demotools.IQuestioner
resources Resources
s3Actions *actions.S3Actions
sdkConfig aws.Config
}
// NewObjectLockScenario constructs a new ObjectLockScenario instance.
func NewObjectLockScenario(sdkConfig aws.Config, questioner demotools.IQuestioner) ObjectLockScenario {
scenario := ObjectLockScenario{
questioner: questioner,
resources: Resources{},
s3Actions: &actions.S3Actions{S3Client: s3.NewFromConfig(sdkConfig)},
sdkConfig: sdkConfig,
}
scenario.s3Actions.S3Manager = manager.NewUploader(scenario.s3Actions.S3Client)
scenario.resources.init(scenario.s3Actions, questioner)
return scenario
}
type nameLocked struct {
name string
locked bool
}
var createInfo = []nameLocked{
{"standard-bucket", false},
{"lock-bucket", true},
{"retention-bucket", false},
}
// CreateBuckets creates the S3 buckets required for the workflow.
func (scenario *ObjectLockScenario) CreateBuckets(ctx context.Context) {
log.Println("Let's create some S3 buckets to use for this workflow.")
success := false
for !success {
prefix := scenario.questioner.Ask(
"This example creates three buckets. Enter a prefix to name your buckets (remember bucket names must be globally unique):")
for _, info := range createInfo {
log.Println(fmt.Sprintf("%s.%s", prefix, info.name))
bucketName, err := scenario.s3Actions.CreateBucketWithLock(ctx, fmt.Sprintf("%s.%s", prefix, info.name), scenario.sdkConfig.Region, info.locked)
if err != nil {
switch err.(type) {
case *types.BucketAlreadyExists, *types.BucketAlreadyOwnedByYou:
log.Printf("Couldn't create bucket %s.\n", bucketName)
default:
panic(err)
}
break
}
scenario.resources.demoBuckets[info.name] = &DemoBucket{
name: bucketName,
objectKeys: []string{},
}
log.Printf("Created bucket %s.\n", bucketName)
}
if len(scenario.resources.demoBuckets) < len(createInfo) {
scenario.resources.deleteBuckets(ctx)
} else {
success = true
}
}
log.Println("S3 buckets created.")
log.Println(strings.Repeat("-", 88))
}
// EnableLockOnBucket enables object locking on an existing bucket.
func (scenario *ObjectLockScenario) EnableLockOnBucket(ctx context.Context) {
log.Println("\nA bucket can be configured to use object locking.")
scenario.questioner.Ask("Press Enter to continue.")
var err error
bucket := scenario.resources.demoBuckets["retention-bucket"]
err = scenario.s3Actions.EnableObjectLockOnBucket(ctx, bucket.name)
if err != nil {
switch err.(type) {
case *types.NoSuchBucket:
log.Printf("Couldn't enable object locking on bucket %s.\n", bucket.name)
default:
panic(err)
}
} else {
log.Printf("Object locking enabled on bucket %s.", bucket.name)
}
log.Println(strings.Repeat("-", 88))
}
// SetDefaultRetentionPolicy sets a default retention governance policy on a bucket.
func (scenario *ObjectLockScenario) SetDefaultRetentionPolicy(ctx context.Context) {
log.Println("\nA bucket can be configured to use object locking with a default retention period.")
bucket := scenario.resources.demoBuckets["retention-bucket"]
retentionPeriod := scenario.questioner.AskInt("Enter the default retention period in days: ")
err := scenario.s3Actions.ModifyDefaultBucketRetention(ctx, bucket.name, types.ObjectLockEnabledEnabled, int32(retentionPeriod), types.ObjectLockRetentionModeGovernance)
if err != nil {
switch err.(type) {
case *types.NoSuchBucket:
log.Printf("Couldn't configure a default retention period on bucket %s.\n", bucket.name)
default:
panic(err)
}
} else {
log.Printf("Default retention policy set on bucket %s with %d day retention period.", bucket.name, retentionPeriod)
bucket.retentionEnabled = true
}
log.Println(strings.Repeat("-", 88))
}
// UploadTestObjects uploads test objects to the S3 buckets.
func (scenario *ObjectLockScenario) UploadTestObjects(ctx context.Context) {
log.Println("Uploading test objects to S3 buckets.")
for _, info := range createInfo {
bucket := scenario.resources.demoBuckets[info.name]
for i := 0; i < 2; i++ {
key, err := scenario.s3Actions.UploadObject(ctx, bucket.name, fmt.Sprintf("example-%d", i),
fmt.Sprintf("Example object content #%d in bucket %s.", i, bucket.name))
if err != nil {
switch err.(type) {
case *types.NoSuchBucket:
log.Printf("Couldn't upload %s to bucket %s.\n", key, bucket.name)
default:
panic(err)
}
} else {
log.Printf("Uploaded %s to bucket %s.\n", key, bucket.name)
bucket.objectKeys = append(bucket.objectKeys, key)
}
}
}
scenario.questioner.Ask("Test objects uploaded. Press Enter to continue.")
log.Println(strings.Repeat("-", 88))
}
// SetObjectLockConfigurations sets object lock configurations on the test objects.
func (scenario *ObjectLockScenario) SetObjectLockConfigurations(ctx context.Context) {
log.Println("Now let's set object lock configurations on individual objects.")
buckets := []*DemoBucket{scenario.resources.demoBuckets["lock-bucket"], scenario.resources.demoBuckets["retention-bucket"]}
for _, bucket := range buckets {
for index, objKey := range bucket.objectKeys {
switch index {
case 0:
if scenario.questioner.AskBool(fmt.Sprintf("\nDo you want to add a legal hold to %s in %s (y/n)? ", objKey, bucket.name), "y") {
err := scenario.s3Actions.PutObjectLegalHold(ctx, bucket.name, objKey, "", types.ObjectLockLegalHoldStatusOn)
if err != nil {
switch err.(type) {
case *types.NoSuchKey:
log.Printf("Couldn't set legal hold on %s.\n", objKey)
default:
panic(err)
}
} else {
log.Printf("Legal hold set on %s.\n", objKey)
}
}
case 1:
q := fmt.Sprintf("\nDo you want to add a 1 day Governance retention period to %s in %s?\n"+
"Reminder: Only a user with the s3:BypassGovernanceRetention permission is able to delete this object\n"+
"or its bucket until the retention period has expired. (y/n) ", objKey, bucket.name)
if scenario.questioner.AskBool(q, "y") {
err := scenario.s3Actions.PutObjectRetention(ctx, bucket.name, objKey, types.ObjectLockRetentionModeGovernance, 1)
if err != nil {
switch err.(type) {
case *types.NoSuchKey:
log.Printf("Couldn't set retention period on %s in %s.\n", objKey, bucket.name)
default:
panic(err)
}
} else {
log.Printf("Retention period set to 1 for %s.", objKey)
bucket.retentionEnabled = true
}
}
}
}
}
log.Println(strings.Repeat("-", 88))
}
const (
ListAll = iota
DeleteObject
DeleteRetentionObject
OverwriteObject
ViewRetention
ViewLegalHold
Finish
)
// InteractWithObjects allows the user to interact with the objects and test the object lock configurations.
func (scenario *ObjectLockScenario) InteractWithObjects(ctx context.Context) {
log.Println("Now you can interact with the objects to explore the object lock configurations.")
interactiveChoices := []string{
"List all objects and buckets.",
"Attempt to delete an object.",
"Attempt to delete an object with retention period bypass.",
"Attempt to overwrite a file.",
"View the retention settings for an object.",
"View the legal hold settings for an object.",
"Finish the workflow."}
choice := ListAll
for choice != Finish {
objList := scenario.GetAllObjects(ctx)
objChoices := scenario.makeObjectChoiceList(objList)
choice = scenario.questioner.AskChoice("Choose an action from the menu:\n", interactiveChoices)
switch choice {
case ListAll:
log.Println("The current objects in the example buckets are:")
for _, objChoice := range objChoices {
log.Println("\t", objChoice)
}
case DeleteObject, DeleteRetentionObject:
objChoice := scenario.questioner.AskChoice("Enter the number of the object to delete:\n", objChoices)
obj := objList[objChoice]
deleted, err := scenario.s3Actions.DeleteObject(ctx, obj.bucket, obj.key, obj.versionId, choice == DeleteRetentionObject)
if err != nil {
switch err.(type) {
case *types.NoSuchKey:
log.Println("Nothing to delete.")
default:
panic(err)
}
} else if deleted {
log.Printf("Object %s deleted.\n", obj.key)
}
case OverwriteObject:
objChoice := scenario.questioner.AskChoice("Enter the number of the object to overwrite:\n", objChoices)
obj := objList[objChoice]
_, err := scenario.s3Actions.UploadObject(ctx, obj.bucket, obj.key, fmt.Sprintf("New content in object %s.", obj.key))
if err != nil {
switch err.(type) {
case *types.NoSuchBucket:
log.Println("Couldn't upload to nonexistent bucket.")
default:
panic(err)
}
} else {
log.Printf("Uploaded new content to object %s.\n", obj.key)
}
case ViewRetention:
objChoice := scenario.questioner.AskChoice("Enter the number of the object to view:\n", objChoices)
obj := objList[objChoice]
retention, err := scenario.s3Actions.GetObjectRetention(ctx, obj.bucket, obj.key)
if err != nil {
switch err.(type) {
case *types.NoSuchKey:
log.Printf("Can't get retention configuration for %s.\n", obj.key)
default:
panic(err)
}
} else if retention != nil {
log.Printf("Object %s has retention mode %s until %v.\n", obj.key, retention.Mode, retention.RetainUntilDate)
} else {
log.Printf("Object %s does not have object retention configured.\n", obj.key)
}
case ViewLegalHold:
objChoice := scenario.questioner.AskChoice("Enter the number of the object to view:\n", objChoices)
obj := objList[objChoice]
legalHold, err := scenario.s3Actions.GetObjectLegalHold(ctx, obj.bucket, obj.key, obj.versionId)
if err != nil {
switch err.(type) {
case *types.NoSuchKey:
log.Printf("Can't get legal hold configuration for %s.\n", obj.key)
default:
panic(err)
}
} else if legalHold != nil {
log.Printf("Object %s has legal hold %v.", obj.key, *legalHold)
} else {
log.Printf("Object %s does not have legal hold configured.", obj.key)
}
case Finish:
log.Println("Let's clean up.")
}
log.Println(strings.Repeat("-", 88))
}
}
type BucketKeyVersionId struct {
bucket string
key string
versionId string
}
// GetAllObjects gets the object versions in the example S3 buckets and returns them in a flattened list.
func (scenario *ObjectLockScenario) GetAllObjects(ctx context.Context) []BucketKeyVersionId {
var objectList []BucketKeyVersionId
for _, info := range createInfo {
bucket := scenario.resources.demoBuckets[info.name]
versions, err := scenario.s3Actions.ListObjectVersions(ctx, bucket.name)
if err != nil {
switch err.(type) {
case *types.NoSuchBucket:
log.Printf("Couldn't get object versions for %s.\n", bucket.name)
default:
panic(err)
}
} else {
for _, version := range versions {
objectList = append(objectList,
BucketKeyVersionId{bucket: bucket.name, key: *version.Key, versionId: *version.VersionId})
}
}
}
return objectList
}
// makeObjectChoiceList makes the object version list into a list of strings that are displayed
// as choices.
func (scenario *ObjectLockScenario) makeObjectChoiceList(bucketObjects []BucketKeyVersionId) []string {
choices := make([]string, len(bucketObjects))
for i := 0; i < len(bucketObjects); i++ {
choices[i] = fmt.Sprintf("%s in %s with VersionId %s.",
bucketObjects[i].key, bucketObjects[i].bucket, bucketObjects[i].versionId)
}
return choices
}
// Run runs the S3 Object Lock scenario.
func (scenario *ObjectLockScenario) Run(ctx context.Context) {
defer func() {
if r := recover(); r != nil {
log.Println("Something went wrong with the demo.")
_, isMock := scenario.questioner.(*demotools.MockQuestioner)
if isMock || scenario.questioner.AskBool("Do you want to see the full error message (y/n)?", "y") {
log.Println(r)
}
scenario.resources.Cleanup(ctx)
}
}()
log.Println(strings.Repeat("-", 88))
log.Println("Welcome to the Amazon S3 Object Lock Feature Scenario.")
log.Println(strings.Repeat("-", 88))
scenario.CreateBuckets(ctx)
scenario.EnableLockOnBucket(ctx)
scenario.SetDefaultRetentionPolicy(ctx)
scenario.UploadTestObjects(ctx)
scenario.SetObjectLockConfigurations(ctx)
scenario.InteractWithObjects(ctx)
scenario.resources.Cleanup(ctx)
log.Println(strings.Repeat("-", 88))
log.Println("Thanks for watching!")
log.Println(strings.Repeat("-", 88))
}
```
Define a struct that wraps S3 actions used in this example.
```
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/aws/smithy-go"
)
// S3Actions wraps S3 service actions.
type S3Actions struct {
S3Client *s3.Client
S3Manager *manager.Uploader
}
// CreateBucketWithLock creates a new S3 bucket with optional object locking enabled
// and waits for the bucket to exist before returning.
func (actor S3Actions) CreateBucketWithLock(ctx context.Context, bucket string, region string, enableObjectLock bool) (string, error) {
input := &s3.CreateBucketInput{
Bucket: aws.String(bucket),
CreateBucketConfiguration: &types.CreateBucketConfiguration{
LocationConstraint: types.BucketLocationConstraint(region),
},
}
if enableObjectLock {
input.ObjectLockEnabledForBucket = aws.Bool(true)
}
_, err := actor.S3Client.CreateBucket(ctx, input)
if err != nil {
var owned *types.BucketAlreadyOwnedByYou
var exists *types.BucketAlreadyExists
if errors.As(err, &owned) {
log.Printf("You already own bucket %s.\n", bucket)
err = owned
} else if errors.As(err, &exists) {
log.Printf("Bucket %s already exists.\n", bucket)
err = exists
}
} else {
err = s3.NewBucketExistsWaiter(actor.S3Client).Wait(
ctx, &s3.HeadBucketInput{Bucket: aws.String(bucket)}, time.Minute)
if err != nil {
log.Printf("Failed attempt to wait for bucket %s to exist.\n", bucket)
}
}
return bucket, err
}
// GetObjectLegalHold retrieves the legal hold status for an S3 object.
func (actor S3Actions) GetObjectLegalHold(ctx context.Context, bucket string, key string, versionId string) (*types.ObjectLockLegalHoldStatus, error) {
var status *types.ObjectLockLegalHoldStatus
input := &s3.GetObjectLegalHoldInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
VersionId: aws.String(versionId),
}
output, err := actor.S3Client.GetObjectLegalHold(ctx, input)
if err != nil {
var noSuchKeyErr *types.NoSuchKey
var apiErr *smithy.GenericAPIError
if errors.As(err, &noSuchKeyErr) {
log.Printf("Object %s does not exist in bucket %s.\n", key, bucket)
err = noSuchKeyErr
} else if errors.As(err, &apiErr) {
switch apiErr.ErrorCode() {
case "NoSuchObjectLockConfiguration":
log.Printf("Object %s does not have an object lock configuration.\n", key)
err = nil
case "InvalidRequest":
log.Printf("Bucket %s does not have an object lock configuration.\n", bucket)
err = nil
}
}
} else {
status = &output.LegalHold.Status
}
return status, err
}
// GetObjectLockConfiguration retrieves the object lock configuration for an S3 bucket.
func (actor S3Actions) GetObjectLockConfiguration(ctx context.Context, bucket string) (*types.ObjectLockConfiguration, error) {
var lockConfig *types.ObjectLockConfiguration
input := &s3.GetObjectLockConfigurationInput{
Bucket: aws.String(bucket),
}
output, err := actor.S3Client.GetObjectLockConfiguration(ctx, input)
if err != nil {
var noBucket *types.NoSuchBucket
var apiErr *smithy.GenericAPIError
if errors.As(err, &noBucket) {
log.Printf("Bucket %s does not exist.\n", bucket)
err = noBucket
} else if errors.As(err, &apiErr) && apiErr.ErrorCode() == "ObjectLockConfigurationNotFoundError" {
log.Printf("Bucket %s does not have an object lock configuration.\n", bucket)
err = nil
}
} else {
lockConfig = output.ObjectLockConfiguration
}
return lockConfig, err
}
// GetObjectRetention retrieves the object retention configuration for an S3 object.
func (actor S3Actions) GetObjectRetention(ctx context.Context, bucket string, key string) (*types.ObjectLockRetention, error) {
var retention *types.ObjectLockRetention
input := &s3.GetObjectRetentionInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
output, err := actor.S3Client.GetObjectRetention(ctx, input)
if err != nil {
var noKey *types.NoSuchKey
var apiErr *smithy.GenericAPIError
if errors.As(err, &noKey) {
log.Printf("Object %s does not exist in bucket %s.\n", key, bucket)
err = noKey
} else if errors.As(err, &apiErr) {
switch apiErr.ErrorCode() {
case "NoSuchObjectLockConfiguration":
err = nil
case "InvalidRequest":
log.Printf("Bucket %s does not have locking enabled.", bucket)
err = nil
}
}
} else {
retention = output.Retention
}
return retention, err
}
// PutObjectLegalHold sets the legal hold configuration for an S3 object.
func (actor S3Actions) PutObjectLegalHold(ctx context.Context, bucket string, key string, versionId string, legalHoldStatus types.ObjectLockLegalHoldStatus) error {
input := &s3.PutObjectLegalHoldInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
LegalHold: &types.ObjectLockLegalHold{
Status: legalHoldStatus,
},
}
if versionId != "" {
input.VersionId = aws.String(versionId)
}
_, err := actor.S3Client.PutObjectLegalHold(ctx, input)
if err != nil {
var noKey *types.NoSuchKey
if errors.As(err, &noKey) {
log.Printf("Object %s does not exist in bucket %s.\n", key, bucket)
err = noKey
}
}
return err
}
// ModifyDefaultBucketRetention modifies the default retention period of an existing bucket.
func (actor S3Actions) ModifyDefaultBucketRetention(
ctx context.Context, bucket string, lockMode types.ObjectLockEnabled, retentionPeriod int32, retentionMode types.ObjectLockRetentionMode) error {
input := &s3.PutObjectLockConfigurationInput{
Bucket: aws.String(bucket),
ObjectLockConfiguration: &types.ObjectLockConfiguration{
ObjectLockEnabled: lockMode,
Rule: &types.ObjectLockRule{
DefaultRetention: &types.DefaultRetention{
Days: aws.Int32(retentionPeriod),
Mode: retentionMode,
},
},
},
}
_, err := actor.S3Client.PutObjectLockConfiguration(ctx, input)
if err != nil {
var noBucket *types.NoSuchBucket
if errors.As(err, &noBucket) {
log.Printf("Bucket %s does not exist.\n", bucket)
err = noBucket
}
}
return err
}
// EnableObjectLockOnBucket enables object locking on an existing bucket.
func (actor S3Actions) EnableObjectLockOnBucket(ctx context.Context, bucket string) error {
// Versioning must be enabled on the bucket before object locking is enabled.
verInput := &s3.PutBucketVersioningInput{
Bucket: aws.String(bucket),
VersioningConfiguration: &types.VersioningConfiguration{
MFADelete: types.MFADeleteDisabled,
Status: types.BucketVersioningStatusEnabled,
},
}
_, err := actor.S3Client.PutBucketVersioning(ctx, verInput)
if err != nil {
var noBucket *types.NoSuchBucket
if errors.As(err, &noBucket) {
log.Printf("Bucket %s does not exist.\n", bucket)
err = noBucket
}
return err
}
input := &s3.PutObjectLockConfigurationInput{
Bucket: aws.String(bucket),
ObjectLockConfiguration: &types.ObjectLockConfiguration{
ObjectLockEnabled: types.ObjectLockEnabledEnabled,
},
}
_, err = actor.S3Client.PutObjectLockConfiguration(ctx, input)
if err != nil {
var noBucket *types.NoSuchBucket
if errors.As(err, &noBucket) {
log.Printf("Bucket %s does not exist.\n", bucket)
err = noBucket
}
}
return err
}
// PutObjectRetention sets the object retention configuration for an S3 object.
func (actor S3Actions) PutObjectRetention(ctx context.Context, bucket string, key string, retentionMode types.ObjectLockRetentionMode, retentionPeriodDays int32) error {
input := &s3.PutObjectRetentionInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
Retention: &types.ObjectLockRetention{
Mode: retentionMode,
RetainUntilDate: aws.Time(time.Now().AddDate(0, 0, int(retentionPeriodDays))),
},
BypassGovernanceRetention: aws.Bool(true),
}
_, err := actor.S3Client.PutObjectRetention(ctx, input)
if err != nil {
var noKey *types.NoSuchKey
if errors.As(err, &noKey) {
log.Printf("Object %s does not exist in bucket %s.\n", key, bucket)
err = noKey
}
}
return err
}
// UploadObject uses the S3 upload manager to upload an object to a bucket.
func (actor S3Actions) UploadObject(ctx context.Context, bucket string, key string, contents string) (string, error) {
var outKey string
input := &s3.PutObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
Body: bytes.NewReader([]byte(contents)),
ChecksumAlgorithm: types.ChecksumAlgorithmSha256,
}
output, err := actor.S3Manager.Upload(ctx, input)
if err != nil {
var noBucket *types.NoSuchBucket
if errors.As(err, &noBucket) {
log.Printf("Bucket %s does not exist.\n", bucket)
err = noBucket
}
} else {
err := s3.NewObjectExistsWaiter(actor.S3Client).Wait(ctx, &s3.HeadObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}, time.Minute)
if err != nil {
log.Printf("Failed attempt to wait for object %s to exist in %s.\n", key, bucket)
} else {
outKey = *output.Key
}
}
return outKey, err
}
// ListObjectVersions lists all versions of all objects in a bucket.
func (actor S3Actions) ListObjectVersions(ctx context.Context, bucket string) ([]types.ObjectVersion, error) {
var err error
var output *s3.ListObjectVersionsOutput
var versions []types.ObjectVersion
input := &s3.ListObjectVersionsInput{Bucket: aws.String(bucket)}
versionPaginator := s3.NewListObjectVersionsPaginator(actor.S3Client, input)
for versionPaginator.HasMorePages() {
output, err = versionPaginator.NextPage(ctx)
if err != nil {
var noBucket *types.NoSuchBucket
if errors.As(err, &noBucket) {
log.Printf("Bucket %s does not exist.\n", bucket)
err = noBucket
}
break
} else {
versions = append(versions, output.Versions...)
}
}
return versions, err
}
// DeleteObject deletes an object from a bucket.
func (actor S3Actions) DeleteObject(ctx context.Context, bucket string, key string, versionId string, bypassGovernance bool) (bool, error) {
deleted := false
input := &s3.DeleteObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
if versionId != "" {
input.VersionId = aws.String(versionId)
}
if bypassGovernance {
input.BypassGovernanceRetention = aws.Bool(true)
}
_, err := actor.S3Client.DeleteObject(ctx, input)
if err != nil {
var noKey *types.NoSuchKey
var apiErr *smithy.GenericAPIError
if errors.As(err, &noKey) {
log.Printf("Object %s does not exist in %s.\n", key, bucket)
err = noKey
} else if errors.As(err, &apiErr) {
switch apiErr.ErrorCode() {
case "AccessDenied":
log.Printf("Access denied: cannot delete object %s from %s.\n", key, bucket)
err = nil
case "InvalidArgument":
if bypassGovernance {
log.Printf("You cannot specify bypass governance on a bucket without lock enabled.")
err = nil
}
}
}
} else {
err = s3.NewObjectNotExistsWaiter(actor.S3Client).Wait(
ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)}, time.Minute)
if err != nil {
log.Printf("Failed attempt to wait for object %s in bucket %s to be deleted.\n", key, bucket)
} else {
deleted = true
}
}
return deleted, err
}
// DeleteObjects deletes a list of objects from a bucket.
func (actor S3Actions) DeleteObjects(ctx context.Context, bucket string, objects []types.ObjectIdentifier, bypassGovernance bool) error {
if len(objects) == 0 {
return nil
}
input := s3.DeleteObjectsInput{
Bucket: aws.String(bucket),
Delete: &types.Delete{
Objects: objects,
Quiet: aws.Bool(true),
},
}
if bypassGovernance {
input.BypassGovernanceRetention = aws.Bool(true)
}
delOut, err := actor.S3Client.DeleteObjects(ctx, &input)
if err != nil || len(delOut.Errors) > 0 {
log.Printf("Error deleting objects from bucket %s.\n", bucket)
if err != nil {
var noBucket *types.NoSuchBucket
if errors.As(err, &noBucket) {
log.Printf("Bucket %s does not exist.\n", bucket)
err = noBucket
}
} else if len(delOut.Errors) > 0 {
for _, outErr := range delOut.Errors {
log.Printf("%s: %s\n", *outErr.Key, *outErr.Message)
}
err = fmt.Errorf("%s", *delOut.Errors[0].Message)
}
} else {
for _, delObjs := range delOut.Deleted {
err = s3.NewObjectNotExistsWaiter(actor.S3Client).Wait(
ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: delObjs.Key}, time.Minute)
if err != nil {
log.Printf("Failed attempt to wait for object %s to be deleted.\n", *delObjs.Key)
} else {
log.Printf("Deleted %s.\n", *delObjs.Key)
}
}
}
return err
}
```
Clean up resources.
```
import (
"context"
"log"
"s3_object_lock/actions"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/awsdocs/aws-doc-sdk-examples/gov2/demotools"
)
// DemoBucket contains metadata for buckets used in this example.
type DemoBucket struct {
name string
retentionEnabled bool
objectKeys []string
}
// Resources keeps track of AWS resources created during the ObjectLockScenario and handles
// cleanup when the scenario finishes.
type Resources struct {
demoBuckets map[string]*DemoBucket
s3Actions *actions.S3Actions
questioner demotools.IQuestioner
}
// init initializes objects in the Resources struct.
func (resources *Resources) init(s3Actions *actions.S3Actions, questioner demotools.IQuestioner) {
resources.s3Actions = s3Actions
resources.questioner = questioner
resources.demoBuckets = map[string]*DemoBucket{}
}
// Cleanup deletes all AWS resources created during the ObjectLockScenario.
func (resources *Resources) Cleanup(ctx context.Context) {
defer func() {
if r := recover(); r != nil {
log.Printf("Something went wrong during cleanup.\n%v\n", r)
log.Println("Use the AWS Management Console to remove any remaining resources " +
"that were created for this scenario.")
}
}()
wantDelete := resources.questioner.AskBool("Do you want to remove all of the AWS resources that were created "+
"during this demo (y/n)?", "y")
if !wantDelete {
log.Println("Be sure to remove resources when you're done with them to avoid unexpected charges!")
return
}
log.Println("Removing objects from S3 buckets and deleting buckets...")
resources.deleteBuckets(ctx)
//resources.deleteRetentionObjects(resources.retentionBucket, resources.retentionObjects)
log.Println("Cleanup complete.")
}
// deleteBuckets empties and then deletes all buckets created during the ObjectLockScenario.
func (resources *Resources) deleteBuckets(ctx context.Context) {
for _, info := range createInfo {
bucket := resources.demoBuckets[info.name]
resources.deleteObjects(ctx, bucket)
_, err := resources.s3Actions.S3Client.DeleteBucket(ctx, &s3.DeleteBucketInput{
Bucket: aws.String(bucket.name),
})
if err != nil {
panic(err)
}
}
for _, info := range createInfo {
bucket := resources.demoBuckets[info.name]
err := s3.NewBucketNotExistsWaiter(resources.s3Actions.S3Client).Wait(
ctx, &s3.HeadBucketInput{Bucket: aws.String(bucket.name)}, time.Minute)
if err != nil {
log.Printf("Failed attempt to wait for bucket %s to be deleted.\n", bucket.name)
} else {
log.Printf("Deleted %s.\n", bucket.name)
}
}
resources.demoBuckets = map[string]*DemoBucket{}
}
// deleteObjects deletes all objects in the specified bucket.
func (resources *Resources) deleteObjects(ctx context.Context, bucket *DemoBucket) {
lockConfig, err := resources.s3Actions.GetObjectLockConfiguration(ctx, bucket.name)
if err != nil {
panic(err)
}
versions, err := resources.s3Actions.ListObjectVersions(ctx, bucket.name)
if err != nil {
switch err.(type) {
case *types.NoSuchBucket:
log.Printf("No objects to get from %s.\n", bucket.name)
default:
panic(err)
}
}
delObjects := make([]types.ObjectIdentifier, len(versions))
for i, version := range versions {
if lockConfig != nil && lockConfig.ObjectLockEnabled == types.ObjectLockEnabledEnabled {
status, err := resources.s3Actions.GetObjectLegalHold(ctx, bucket.name, *version.Key, *version.VersionId)
if err != nil {
switch err.(type) {
case *types.NoSuchKey:
log.Printf("Couldn't determine legal hold status for %s in %s.\n", *version.Key, bucket.name)
default:
panic(err)
}
} else if status != nil && *status == types.ObjectLockLegalHoldStatusOn {
err = resources.s3Actions.PutObjectLegalHold(ctx, bucket.name, *version.Key, *version.VersionId, types.ObjectLockLegalHoldStatusOff)
if err != nil {
switch err.(type) {
case *types.NoSuchKey:
log.Printf("Couldn't turn off legal hold for %s in %s.\n", *version.Key, bucket.name)
default:
panic(err)
}
}
}
}
delObjects[i] = types.ObjectIdentifier{Key: version.Key, VersionId: version.VersionId}
}
err = resources.s3Actions.DeleteObjects(ctx, bucket.name, delObjects, bucket.retentionEnabled)
if err != nil {
switch err.(type) {
case *types.NoSuchBucket:
log.Println("Nothing to delete.")
default:
panic(err)
}
}
}
```
+ For API details, see the following topics in *AWS SDK for Go API Reference*.
+ [GetObjectLegalHold](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3#Client.GetObjectLegalHold)
+ [GetObjectLockConfiguration](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3#Client.GetObjectLockConfiguration)
+ [GetObjectRetention](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3#Client.GetObjectRetention)
+ [PutObjectLegalHold](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3#Client.PutObjectLegalHold)
+ [PutObjectLockConfiguration](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3#Client.PutObjectLockConfiguration)
+ [PutObjectRetention](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3#Client.PutObjectRetention)
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3/src/main/java/com/example/s3/lockscenario#code-examples).
Run an interactive scenario demonstrating Amazon S3 object lock features.
```
import software.amazon.awssdk.services.s3.model.ObjectLockLegalHold;
import software.amazon.awssdk.services.s3.model.ObjectLockRetention;
import java.io.BufferedWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import java.util.stream.Collectors;
/*
Before running this Java V2 code example, set up your development
environment, including your credentials.
For more information, see the following documentation topic:
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/setup.html
This Java example performs the following tasks:
1. Create test Amazon Simple Storage Service (S3) buckets with different lock policies.
2. Upload sample objects to each bucket.
3. Set some Legal Hold and Retention Periods on objects and buckets.
4. Investigate lock policies by viewing settings or attempting to delete or overwrite objects.
5. Clean up objects and buckets.
*/
public class S3ObjectLockWorkflow {
public static final String DASHES = new String(new char[80]).replace("\0", "-");
static String bucketName;
static S3LockActions s3LockActions;
private static final List bucketNames = new ArrayList<>();
private static final List fileNames = new ArrayList<>();
public static void main(String[] args) {
final String usage = """
Usage:
\s
Where:
bucketName - The Amazon S3 bucket name.
""";
if (args.length != 1) {
System.out.println(usage);
System.exit(1);
}
s3LockActions = new S3LockActions();
bucketName = args[0];
Scanner scanner = new Scanner(System.in);
System.out.println(DASHES);
System.out.println("Welcome to the Amazon Simple Storage Service (S3) Object Locking Feature Scenario.");
System.out.println("Press Enter to continue...");
scanner.nextLine();
configurationSetup();
System.out.println(DASHES);
System.out.println(DASHES);
setup();
System.out.println("Setup is complete. Press Enter to continue...");
scanner.nextLine();
System.out.println(DASHES);
System.out.println(DASHES);
System.out.println("Lets present the user with choices.");
System.out.println("Press Enter to continue...");
scanner.nextLine();
demoActionChoices() ;
System.out.println(DASHES);
System.out.println(DASHES);
System.out.println("Would you like to clean up the resources? (y/n)");
String delAns = scanner.nextLine().trim();
if (delAns.equalsIgnoreCase("y")) {
cleanup();
System.out.println("Clean up is complete.");
}
System.out.println("Press Enter to continue...");
scanner.nextLine();
System.out.println(DASHES);
System.out.println(DASHES);
System.out.println("Amazon S3 Object Locking Workflow is complete.");
System.out.println(DASHES);
}
// Present the user with the demo action choices.
public static void demoActionChoices() {
String[] choices = {
"List all files in buckets.",
"Attempt to delete a file.",
"Attempt to delete a file with retention period bypass.",
"Attempt to overwrite a file.",
"View the object and bucket retention settings for a file.",
"View the legal hold settings for a file.",
"Finish the workflow."
};
int choice = 0;
while (true) {
System.out.println(DASHES);
choice = getChoiceResponse("Explore the S3 locking features by selecting one of the following choices:", choices);
System.out.println(DASHES);
System.out.println("You selected "+choices[choice]);
switch (choice) {
case 0 -> {
s3LockActions.listBucketsAndObjects(bucketNames, true);
}
case 1 -> {
System.out.println("Enter the number of the object to delete:");
List allFiles = s3LockActions.listBucketsAndObjects(bucketNames, true);
List fileKeys = allFiles.stream().map(f -> f.getKeyName()).collect(Collectors.toList());
String[] fileKeysArray = fileKeys.toArray(new String[0]);
int fileChoice = getChoiceResponse(null, fileKeysArray);
String objectKey = fileKeys.get(fileChoice);
String bucketName = allFiles.get(fileChoice).getBucketName();
String version = allFiles.get(fileChoice).getVersion();
s3LockActions.deleteObjectFromBucket(bucketName, objectKey, false, version);
}
case 2 -> {
System.out.println("Enter the number of the object to delete:");
List allFiles = s3LockActions.listBucketsAndObjects(bucketNames, true);
List fileKeys = allFiles.stream().map(f -> f.getKeyName()).collect(Collectors.toList());
String[] fileKeysArray = fileKeys.toArray(new String[0]);
int fileChoice = getChoiceResponse(null, fileKeysArray);
String objectKey = fileKeys.get(fileChoice);
String bucketName = allFiles.get(fileChoice).getBucketName();
String version = allFiles.get(fileChoice).getVersion();
s3LockActions.deleteObjectFromBucket(bucketName, objectKey, true, version);
}
case 3 -> {
System.out.println("Enter the number of the object to overwrite:");
List allFiles = s3LockActions.listBucketsAndObjects(bucketNames, true);
List fileKeys = allFiles.stream().map(f -> f.getKeyName()).collect(Collectors.toList());
String[] fileKeysArray = fileKeys.toArray(new String[0]);
int fileChoice = getChoiceResponse(null, fileKeysArray);
String objectKey = fileKeys.get(fileChoice);
String bucketName = allFiles.get(fileChoice).getBucketName();
// Attempt to overwrite the file.
try (BufferedWriter writer = new BufferedWriter(new java.io.FileWriter(objectKey))) {
writer.write("This is a modified text.");
} catch (IOException e) {
e.printStackTrace();
}
s3LockActions.uploadFile(bucketName, objectKey, objectKey);
}
case 4 -> {
System.out.println("Enter the number of the object to overwrite:");
List allFiles = s3LockActions.listBucketsAndObjects(bucketNames, true);
List fileKeys = allFiles.stream().map(f -> f.getKeyName()).collect(Collectors.toList());
String[] fileKeysArray = fileKeys.toArray(new String[0]);
int fileChoice = getChoiceResponse(null, fileKeysArray);
String objectKey = fileKeys.get(fileChoice);
String bucketName = allFiles.get(fileChoice).getBucketName();
s3LockActions.getObjectRetention(bucketName, objectKey);
}
case 5 -> {
System.out.println("Enter the number of the object to view:");
List allFiles = s3LockActions.listBucketsAndObjects(bucketNames, true);
List fileKeys = allFiles.stream().map(f -> f.getKeyName()).collect(Collectors.toList());
String[] fileKeysArray = fileKeys.toArray(new String[0]);
int fileChoice = getChoiceResponse(null, fileKeysArray);
String objectKey = fileKeys.get(fileChoice);
String bucketName = allFiles.get(fileChoice).getBucketName();
s3LockActions.getObjectLegalHold(bucketName, objectKey);
s3LockActions.getBucketObjectLockConfiguration(bucketName);
}
case 6 -> {
System.out.println("Exiting the workflow...");
return;
}
default -> {
System.out.println("Invalid choice. Please select again.");
}
}
}
}
// Clean up the resources from the scenario.
private static void cleanup() {
List allFiles = s3LockActions.listBucketsAndObjects(bucketNames, false);
for (S3InfoObject fileInfo : allFiles) {
String bucketName = fileInfo.getBucketName();
String key = fileInfo.getKeyName();
String version = fileInfo.getVersion();
if (bucketName.contains("lock-enabled") || (bucketName.contains("retention-after-creation"))) {
ObjectLockLegalHold legalHold = s3LockActions.getObjectLegalHold(bucketName, key);
if (legalHold != null) {
String holdStatus = legalHold.status().name();
System.out.println(holdStatus);
if (holdStatus.compareTo("ON") == 0) {
s3LockActions.modifyObjectLegalHold(bucketName, key, false);
}
}
// Check for a retention period.
ObjectLockRetention retention = s3LockActions.getObjectRetention(bucketName, key);
boolean hasRetentionPeriod ;
hasRetentionPeriod = retention != null;
s3LockActions.deleteObjectFromBucket(bucketName, key,hasRetentionPeriod, version);
} else {
System.out.println(bucketName +" objects do not have a legal lock");
s3LockActions.deleteObjectFromBucket(bucketName, key,false, version);
}
}
// Delete the buckets.
System.out.println("Delete "+bucketName);
for (String bucket : bucketNames){
s3LockActions.deleteBucketByName(bucket);
}
}
private static void setup() {
Scanner scanner = new Scanner(System.in);
System.out.println("""
For this workflow, we will use the AWS SDK for Java to create several S3
buckets and files to demonstrate working with S3 locking features.
""");
System.out.println("S3 buckets can be created either with or without object lock enabled.");
System.out.println("Press Enter to continue...");
scanner.nextLine();
// Create three S3 buckets.
s3LockActions.createBucketWithLockOptions(false, bucketNames.get(0));
s3LockActions.createBucketWithLockOptions(true, bucketNames.get(1));
s3LockActions.createBucketWithLockOptions(false, bucketNames.get(2));
System.out.println("Press Enter to continue.");
scanner.nextLine();
System.out.println("Bucket "+bucketNames.get(2) +" will be configured to use object locking with a default retention period.");
s3LockActions.modifyBucketDefaultRetention(bucketNames.get(2));
System.out.println("Press Enter to continue.");
scanner.nextLine();
System.out.println("Object lock policies can also be added to existing buckets. For this example, we will use "+bucketNames.get(1));
s3LockActions.enableObjectLockOnBucket(bucketNames.get(1));
System.out.println("Press Enter to continue.");
scanner.nextLine();
// Upload some files to the buckets.
System.out.println("Now let's add some test files:");
String fileName = "exampleFile.txt";
int fileCount = 2;
try (BufferedWriter writer = new BufferedWriter(new java.io.FileWriter(fileName))) {
writer.write("This is a sample file for uploading to a bucket.");
} catch (IOException e) {
e.printStackTrace();
}
for (String bucketName : bucketNames){
for (int i = 0; i < fileCount; i++) {
// Get the file name without extension.
String fileNameWithoutExtension = java.nio.file.Paths.get(fileName).getFileName().toString();
int extensionIndex = fileNameWithoutExtension.lastIndexOf('.');
if (extensionIndex > 0) {
fileNameWithoutExtension = fileNameWithoutExtension.substring(0, extensionIndex);
}
// Create the numbered file names.
String numberedFileName = fileNameWithoutExtension + i + getFileExtension(fileName);
fileNames.add(numberedFileName);
s3LockActions.uploadFile(bucketName, numberedFileName, fileName);
}
}
String question = null;
System.out.print("Press Enter to continue...");
scanner.nextLine();
System.out.println("Now we can set some object lock policies on individual files:");
for (String bucketName : bucketNames) {
for (int i = 0; i < fileNames.size(); i++){
// No modifications to the objects in the first bucket.
if (!bucketName.equals(bucketNames.get(0))) {
String exampleFileName = fileNames.get(i);
switch (i) {
case 0 -> {
question = "Would you like to add a legal hold to " + exampleFileName + " in " + bucketName + " (y/n)?";
System.out.println(question);
String ans = scanner.nextLine().trim();
if (ans.equalsIgnoreCase("y")) {
System.out.println("**** You have selected to put a legal hold " + exampleFileName);
// Set a legal hold.
s3LockActions.modifyObjectLegalHold(bucketName, exampleFileName, true);
}
}
case 1 -> {
"""
Would you like to add a 1 day Governance retention period to %s in %s (y/n)?
Reminder: Only a user with the s3:BypassGovernanceRetention permission will be able to delete this file or its bucket until the retention period has expired.
""".formatted(exampleFileName, bucketName);
System.out.println(question);
String ans2 = scanner.nextLine().trim();
if (ans2.equalsIgnoreCase("y")) {
s3LockActions.modifyObjectRetentionPeriod(bucketName, exampleFileName);
}
}
}
}
}
}
}
// Get file extension.
private static String getFileExtension(String fileName) {
int dotIndex = fileName.lastIndexOf('.');
if (dotIndex > 0) {
return fileName.substring(dotIndex);
}
return "";
}
public static void configurationSetup() {
String noLockBucketName = bucketName + "-no-lock";
String lockEnabledBucketName = bucketName + "-lock-enabled";
String retentionAfterCreationBucketName = bucketName + "-retention-after-creation";
bucketNames.add(noLockBucketName);
bucketNames.add(lockEnabledBucketName);
bucketNames.add(retentionAfterCreationBucketName);
}
public static int getChoiceResponse(String question, String[] choices) {
Scanner scanner = new Scanner(System.in);
if (question != null) {
System.out.println(question);
for (int i = 0; i < choices.length; i++) {
System.out.println("\t" + (i + 1) + ". " + choices[i]);
}
}
int choiceNumber = 0;
while (choiceNumber < 1 || choiceNumber > choices.length) {
String choice = scanner.nextLine();
try {
choiceNumber = Integer.parseInt(choice);
} catch (NumberFormatException e) {
System.out.println("Invalid choice. Please enter a valid number.");
}
}
return choiceNumber - 1;
}
}
```
A wrapper class for S3 functions.
```
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.BucketVersioningStatus;
import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.DefaultRetention;
import software.amazon.awssdk.services.s3.model.DeleteBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectLegalHoldRequest;
import software.amazon.awssdk.services.s3.model.GetObjectLegalHoldResponse;
import software.amazon.awssdk.services.s3.model.GetObjectLockConfigurationRequest;
import software.amazon.awssdk.services.s3.model.GetObjectLockConfigurationResponse;
import software.amazon.awssdk.services.s3.model.GetObjectRetentionRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRetentionResponse;
import software.amazon.awssdk.services.s3.model.HeadBucketRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsResponse;
import software.amazon.awssdk.services.s3.model.MFADelete;
import software.amazon.awssdk.services.s3.model.ObjectLockConfiguration;
import software.amazon.awssdk.services.s3.model.ObjectLockEnabled;
import software.amazon.awssdk.services.s3.model.ObjectLockLegalHold;
import software.amazon.awssdk.services.s3.model.ObjectLockLegalHoldStatus;
import software.amazon.awssdk.services.s3.model.ObjectLockRetention;
import software.amazon.awssdk.services.s3.model.ObjectLockRetentionMode;
import software.amazon.awssdk.services.s3.model.ObjectLockRule;
import software.amazon.awssdk.services.s3.model.PutBucketVersioningRequest;
import software.amazon.awssdk.services.s3.model.PutObjectLegalHoldRequest;
import software.amazon.awssdk.services.s3.model.PutObjectLockConfigurationRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRetentionRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.VersioningConfiguration;
import software.amazon.awssdk.services.s3.waiters.S3Waiter;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.time.temporal.ChronoUnit;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
// Contains application logic for the Amazon S3 operations used in this workflow.
public class S3LockActions {
private static S3Client getClient() {
return S3Client.builder()
.region(Region.US_EAST_1)
.build();
}
// Set or modify a retention period on an object in an S3 bucket.
public void modifyObjectRetentionPeriod(String bucketName, String objectKey) {
// Calculate the instant one day from now.
Instant futureInstant = Instant.now().plus(1, ChronoUnit.DAYS);
// Convert the Instant to a ZonedDateTime object with a specific time zone.
ZonedDateTime zonedDateTime = futureInstant.atZone(ZoneId.systemDefault());
// Define a formatter for human-readable output.
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
// Format the ZonedDateTime object to a human-readable date string.
String humanReadableDate = formatter.format(zonedDateTime);
// Print the formatted date string.
System.out.println("Formatted Date: " + humanReadableDate);
ObjectLockRetention retention = ObjectLockRetention.builder()
.mode(ObjectLockRetentionMode.GOVERNANCE)
.retainUntilDate(futureInstant)
.build();
PutObjectRetentionRequest retentionRequest = PutObjectRetentionRequest.builder()
.bucket(bucketName)
.key(objectKey)
.retention(retention)
.build();
getClient().putObjectRetention(retentionRequest);
System.out.println("Set retention for "+objectKey +" in " +bucketName +" until "+ humanReadableDate +".");
}
// Get the legal hold details for an S3 object.
public ObjectLockLegalHold getObjectLegalHold(String bucketName, String objectKey) {
try {
GetObjectLegalHoldRequest legalHoldRequest = GetObjectLegalHoldRequest.builder()
.bucket(bucketName)
.key(objectKey)
.build();
GetObjectLegalHoldResponse response = getClient().getObjectLegalHold(legalHoldRequest);
System.out.println("Object legal hold for " + objectKey + " in " + bucketName +
":\n\tStatus: " + response.legalHold().status());
return response.legalHold();
} catch (S3Exception ex) {
System.out.println("\tUnable to fetch legal hold: '" + ex.getMessage() + "'");
}
return null;
}
// Create a new Amazon S3 bucket with object lock options.
public void createBucketWithLockOptions(boolean enableObjectLock, String bucketName) {
S3Waiter s3Waiter = getClient().waiter();
CreateBucketRequest bucketRequest = CreateBucketRequest.builder()
.bucket(bucketName)
.objectLockEnabledForBucket(enableObjectLock)
.build();
getClient().createBucket(bucketRequest);
HeadBucketRequest bucketRequestWait = HeadBucketRequest.builder()
.bucket(bucketName)
.build();
// Wait until the bucket is created and print out the response.
s3Waiter.waitUntilBucketExists(bucketRequestWait);
System.out.println(bucketName + " is ready");
}
public List listBucketsAndObjects(List bucketNames, Boolean interactive) {
AtomicInteger counter = new AtomicInteger(0); // Initialize counter.
return bucketNames.stream()
.flatMap(bucketName -> listBucketObjectsAndVersions(bucketName).versions().stream()
.map(version -> {
S3InfoObject s3InfoObject = new S3InfoObject();
s3InfoObject.setBucketName(bucketName);
s3InfoObject.setVersion(version.versionId());
s3InfoObject.setKeyName(version.key());
return s3InfoObject;
}))
.peek(s3InfoObject -> {
int i = counter.incrementAndGet(); // Increment and get the updated value.
if (interactive) {
System.out.println(i + ": "+ s3InfoObject.getKeyName());
System.out.printf("%5s Bucket name: %s\n", "", s3InfoObject.getBucketName());
System.out.printf("%5s Version: %s\n", "", s3InfoObject.getVersion());
}
})
.collect(Collectors.toList());
}
public ListObjectVersionsResponse listBucketObjectsAndVersions(String bucketName) {
ListObjectVersionsRequest versionsRequest = ListObjectVersionsRequest.builder()
.bucket(bucketName)
.build();
return getClient().listObjectVersions(versionsRequest);
}
// Set or modify a retention period on an S3 bucket.
public void modifyBucketDefaultRetention(String bucketName) {
VersioningConfiguration versioningConfiguration = VersioningConfiguration.builder()
.mfaDelete(MFADelete.DISABLED)
.status(BucketVersioningStatus.ENABLED)
.build();
PutBucketVersioningRequest versioningRequest = PutBucketVersioningRequest.builder()
.bucket(bucketName)
.versioningConfiguration(versioningConfiguration)
.build();
getClient().putBucketVersioning(versioningRequest);
DefaultRetention rention = DefaultRetention.builder()
.days(1)
.mode(ObjectLockRetentionMode.GOVERNANCE)
.build();
ObjectLockRule lockRule = ObjectLockRule.builder()
.defaultRetention(rention)
.build();
ObjectLockConfiguration objectLockConfiguration = ObjectLockConfiguration.builder()
.objectLockEnabled(ObjectLockEnabled.ENABLED)
.rule(lockRule)
.build();
PutObjectLockConfigurationRequest putObjectLockConfigurationRequest = PutObjectLockConfigurationRequest.builder()
.bucket(bucketName)
.objectLockConfiguration(objectLockConfiguration)
.build();
getClient().putObjectLockConfiguration(putObjectLockConfigurationRequest) ;
System.out.println("Added a default retention to bucket "+bucketName +".");
}
// Enable object lock on an existing bucket.
public void enableObjectLockOnBucket(String bucketName) {
try {
VersioningConfiguration versioningConfiguration = VersioningConfiguration.builder()
.status(BucketVersioningStatus.ENABLED)
.build();
PutBucketVersioningRequest putBucketVersioningRequest = PutBucketVersioningRequest.builder()
.bucket(bucketName)
.versioningConfiguration(versioningConfiguration)
.build();
// Enable versioning on the bucket.
getClient().putBucketVersioning(putBucketVersioningRequest);
PutObjectLockConfigurationRequest request = PutObjectLockConfigurationRequest.builder()
.bucket(bucketName)
.objectLockConfiguration(ObjectLockConfiguration.builder()
.objectLockEnabled(ObjectLockEnabled.ENABLED)
.build())
.build();
getClient().putObjectLockConfiguration(request);
System.out.println("Successfully enabled object lock on "+bucketName);
} catch (S3Exception ex) {
System.out.println("Error modifying object lock: '" + ex.getMessage() + "'");
}
}
public void uploadFile(String bucketName, String objectName, String filePath) {
Path file = Paths.get(filePath);
PutObjectRequest request = PutObjectRequest.builder()
.bucket(bucketName)
.key(objectName)
.checksumAlgorithm(ChecksumAlgorithm.SHA256)
.build();
PutObjectResponse response = getClient().putObject(request, file);
if (response != null) {
System.out.println("\tSuccessfully uploaded " + objectName + " to " + bucketName + ".");
} else {
System.out.println("\tCould not upload " + objectName + " to " + bucketName + ".");
}
}
// Set or modify a legal hold on an object in an S3 bucket.
public void modifyObjectLegalHold(String bucketName, String objectKey, boolean legalHoldOn) {
ObjectLockLegalHold legalHold ;
if (legalHoldOn) {
legalHold = ObjectLockLegalHold.builder()
.status(ObjectLockLegalHoldStatus.ON)
.build();
} else {
legalHold = ObjectLockLegalHold.builder()
.status(ObjectLockLegalHoldStatus.OFF)
.build();
}
PutObjectLegalHoldRequest legalHoldRequest = PutObjectLegalHoldRequest.builder()
.bucket(bucketName)
.key(objectKey)
.legalHold(legalHold)
.build();
getClient().putObjectLegalHold(legalHoldRequest) ;
System.out.println("Modified legal hold for "+ objectKey +" in "+bucketName +".");
}
// Delete an object from a specific bucket.
public void deleteObjectFromBucket(String bucketName, String objectKey, boolean hasRetention, String versionId) {
try {
DeleteObjectRequest objectRequest;
if (hasRetention) {
objectRequest = DeleteObjectRequest.builder()
.bucket(bucketName)
.key(objectKey)
.versionId(versionId)
.bypassGovernanceRetention(true)
.build();
} else {
objectRequest = DeleteObjectRequest.builder()
.bucket(bucketName)
.key(objectKey)
.versionId(versionId)
.build();
}
getClient().deleteObject(objectRequest) ;
System.out.println("The object was successfully deleted");
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
}
}
// Get the retention period for an S3 object.
public ObjectLockRetention getObjectRetention(String bucketName, String key){
try {
GetObjectRetentionRequest retentionRequest = GetObjectRetentionRequest.builder()
.bucket(bucketName)
.key(key)
.build();
GetObjectRetentionResponse response = getClient().getObjectRetention(retentionRequest);
System.out.println("tObject retention for "+key +" in "+ bucketName +": " + response.retention().mode() +" until "+ response.retention().retainUntilDate() +".");
return response.retention();
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
return null;
}
}
public void deleteBucketByName(String bucketName) {
try {
DeleteBucketRequest request = DeleteBucketRequest.builder()
.bucket(bucketName)
.build();
getClient().deleteBucket(request);
System.out.println(bucketName +" was deleted.");
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
}
}
// Get the object lock configuration details for an S3 bucket.
public void getBucketObjectLockConfiguration(String bucketName) {
GetObjectLockConfigurationRequest objectLockConfigurationRequest = GetObjectLockConfigurationRequest.builder()
.bucket(bucketName)
.build();
GetObjectLockConfigurationResponse response = getClient().getObjectLockConfiguration(objectLockConfigurationRequest);
System.out.println("Bucket object lock config for "+bucketName +": ");
System.out.println("\tEnabled: "+response.objectLockConfiguration().objectLockEnabled());
System.out.println("\tRule: "+ response.objectLockConfiguration().rule().defaultRetention());
}
}
```
+ For API details, see the following topics in *AWS SDK for Java 2.x API Reference*.
+ [GetObjectLegalHold](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/GetObjectLegalHold)
+ [GetObjectLockConfiguration](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/GetObjectLockConfiguration)
+ [GetObjectRetention](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/GetObjectRetention)
+ [PutObjectLegalHold](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutObjectLegalHold)
+ [PutObjectLockConfiguration](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutObjectLockConfiguration)
+ [PutObjectRetention](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutObjectRetention)
------
#### [ JavaScript ]
**SDK for JavaScript (v3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javascriptv3/example_code/s3/scenarios/object-locking#code-examples).
Entrypoint for the scenario (index.js). This orchestrates all of the steps. Visit GitHub to see the implementation details for Scenario, ScenarioInput, ScenarioOutput, and ScenarioAction.
```
import * as Scenarios from "@aws-doc-sdk-examples/lib/scenario/index.js";
import {
exitOnFalse,
loadState,
saveState,
} from "@aws-doc-sdk-examples/lib/scenario/steps-common.js";
import { welcome, welcomeContinue } from "./welcome.steps.js";
import {
confirmCreateBuckets,
confirmPopulateBuckets,
confirmSetLegalHoldFileEnabled,
confirmSetLegalHoldFileRetention,
confirmSetRetentionPeriodFileEnabled,
confirmSetRetentionPeriodFileRetention,
confirmUpdateLockPolicy,
confirmUpdateRetention,
createBuckets,
createBucketsAction,
getBucketPrefix,
populateBuckets,
populateBucketsAction,
setLegalHoldFileEnabledAction,
setLegalHoldFileRetentionAction,
setRetentionPeriodFileEnabledAction,
setRetentionPeriodFileRetentionAction,
updateLockPolicy,
updateLockPolicyAction,
updateRetention,
updateRetentionAction,
} from "./setup.steps.js";
/**
* @param {Scenarios} scenarios
* @param {Record} initialState
*/
export const getWorkflowStages = (scenarios, initialState = {}) => {
const client = new S3Client({});
return {
deploy: new scenarios.Scenario(
"S3 Object Locking - Deploy",
[
welcome(scenarios),
welcomeContinue(scenarios),
exitOnFalse(scenarios, "welcomeContinue"),
getBucketPrefix(scenarios),
createBuckets(scenarios),
confirmCreateBuckets(scenarios),
exitOnFalse(scenarios, "confirmCreateBuckets"),
createBucketsAction(scenarios, client),
updateRetention(scenarios),
confirmUpdateRetention(scenarios),
exitOnFalse(scenarios, "confirmUpdateRetention"),
updateRetentionAction(scenarios, client),
populateBuckets(scenarios),
confirmPopulateBuckets(scenarios),
exitOnFalse(scenarios, "confirmPopulateBuckets"),
populateBucketsAction(scenarios, client),
updateLockPolicy(scenarios),
confirmUpdateLockPolicy(scenarios),
exitOnFalse(scenarios, "confirmUpdateLockPolicy"),
updateLockPolicyAction(scenarios, client),
confirmSetLegalHoldFileEnabled(scenarios),
setLegalHoldFileEnabledAction(scenarios, client),
confirmSetRetentionPeriodFileEnabled(scenarios),
setRetentionPeriodFileEnabledAction(scenarios, client),
confirmSetLegalHoldFileRetention(scenarios),
setLegalHoldFileRetentionAction(scenarios, client),
confirmSetRetentionPeriodFileRetention(scenarios),
setRetentionPeriodFileRetentionAction(scenarios, client),
saveState,
],
initialState,
),
demo: new scenarios.Scenario(
"S3 Object Locking - Demo",
[loadState, replAction(scenarios, client)],
initialState,
),
clean: new scenarios.Scenario(
"S3 Object Locking - Destroy",
[
loadState,
confirmCleanup(scenarios),
exitOnFalse(scenarios, "confirmCleanup"),
cleanupAction(scenarios, client),
],
initialState,
),
};
};
// Call function if run directly
import { fileURLToPath } from "node:url";
import { S3Client } from "@aws-sdk/client-s3";
import { cleanupAction, confirmCleanup } from "./clean.steps.js";
import { replAction } from "./repl.steps.js";
if (process.argv[1] === fileURLToPath(import.meta.url)) {
const objectLockingScenarios = getWorkflowStages(Scenarios);
Scenarios.parseScenarioArgs(objectLockingScenarios, {
name: "Amazon S3 object locking workflow",
description:
"Work with Amazon Simple Storage Service (Amazon S3) object locking features.",
synopsis:
"node index.js --scenario [-h|--help] [-y|--yes] [-v|--verbose]",
});
}
```
Output welcome messages to the console (welcome.steps.js).
```
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @param {Scenarios} scenarios
*/
const welcome = (scenarios) =>
new scenarios.ScenarioOutput(
"welcome",
"Welcome to the Amazon Simple Storage Service (S3) Object Locking Feature Scenario. For this workflow, we will use the AWS SDK for JavaScript to create several S3 buckets and files to demonstrate working with S3 locking features.",
{ header: true },
);
/**
* @param {Scenarios} scenarios
*/
const welcomeContinue = (scenarios) =>
new scenarios.ScenarioInput(
"welcomeContinue",
"Press Enter when you are ready to start.",
{ type: "confirm" },
);
export { welcome, welcomeContinue };
```
Deploy buckets, objects, and file settings (setup.steps.js).
```
import {
BucketVersioningStatus,
ChecksumAlgorithm,
CreateBucketCommand,
MFADeleteStatus,
PutBucketVersioningCommand,
PutObjectCommand,
PutObjectLockConfigurationCommand,
PutObjectLegalHoldCommand,
PutObjectRetentionCommand,
ObjectLockLegalHoldStatus,
ObjectLockRetentionMode,
GetBucketVersioningCommand,
BucketAlreadyExists,
BucketAlreadyOwnedByYou,
S3ServiceException,
waitUntilBucketExists,
} from "@aws-sdk/client-s3";
import { retry } from "@aws-doc-sdk-examples/lib/utils/util-timers.js";
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @typedef {import("@aws-sdk/client-s3").S3Client} S3Client
*/
/**
* @param {Scenarios} scenarios
*/
const getBucketPrefix = (scenarios) =>
new scenarios.ScenarioInput(
"bucketPrefix",
"Provide a prefix that will be used for bucket creation.",
{ type: "input", default: "amzn-s3-demo-bucket" },
);
/**
* @param {Scenarios} scenarios
*/
const createBuckets = (scenarios) =>
new scenarios.ScenarioOutput(
"createBuckets",
(state) => `The following buckets will be created:
${state.bucketPrefix}-no-lock with object lock False.
${state.bucketPrefix}-lock-enabled with object lock True.
${state.bucketPrefix}-retention-after-creation with object lock False.`,
{ preformatted: true },
);
/**
* @param {Scenarios} scenarios
*/
const confirmCreateBuckets = (scenarios) =>
new scenarios.ScenarioInput("confirmCreateBuckets", "Create the buckets?", {
type: "confirm",
});
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const createBucketsAction = (scenarios, client) =>
new scenarios.ScenarioAction("createBucketsAction", async (state) => {
const noLockBucketName = `${state.bucketPrefix}-no-lock`;
const lockEnabledBucketName = `${state.bucketPrefix}-lock-enabled`;
const retentionBucketName = `${state.bucketPrefix}-retention-after-creation`;
try {
await client.send(new CreateBucketCommand({ Bucket: noLockBucketName }));
await waitUntilBucketExists({ client }, { Bucket: noLockBucketName });
await client.send(
new CreateBucketCommand({
Bucket: lockEnabledBucketName,
ObjectLockEnabledForBucket: true,
}),
);
await waitUntilBucketExists(
{ client },
{ Bucket: lockEnabledBucketName },
);
await client.send(
new CreateBucketCommand({ Bucket: retentionBucketName }),
);
await waitUntilBucketExists({ client }, { Bucket: retentionBucketName });
state.noLockBucketName = noLockBucketName;
state.lockEnabledBucketName = lockEnabledBucketName;
state.retentionBucketName = retentionBucketName;
} catch (caught) {
if (
caught instanceof BucketAlreadyExists ||
caught instanceof BucketAlreadyOwnedByYou
) {
console.error(`${caught.name}: ${caught.message}`);
state.earlyExit = true;
} else {
throw caught;
}
}
});
/**
* @param {Scenarios} scenarios
*/
const populateBuckets = (scenarios) =>
new scenarios.ScenarioOutput(
"populateBuckets",
(state) => `The following test files will be created:
file0.txt in ${state.bucketPrefix}-no-lock.
file1.txt in ${state.bucketPrefix}-no-lock.
file0.txt in ${state.bucketPrefix}-lock-enabled.
file1.txt in ${state.bucketPrefix}-lock-enabled.
file0.txt in ${state.bucketPrefix}-retention-after-creation.
file1.txt in ${state.bucketPrefix}-retention-after-creation.`,
{ preformatted: true },
);
/**
* @param {Scenarios} scenarios
*/
const confirmPopulateBuckets = (scenarios) =>
new scenarios.ScenarioInput(
"confirmPopulateBuckets",
"Populate the buckets?",
{ type: "confirm" },
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const populateBucketsAction = (scenarios, client) =>
new scenarios.ScenarioAction("populateBucketsAction", async (state) => {
try {
await client.send(
new PutObjectCommand({
Bucket: state.noLockBucketName,
Key: "file0.txt",
Body: "Content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
await client.send(
new PutObjectCommand({
Bucket: state.noLockBucketName,
Key: "file1.txt",
Body: "Content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
await client.send(
new PutObjectCommand({
Bucket: state.lockEnabledBucketName,
Key: "file0.txt",
Body: "Content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
await client.send(
new PutObjectCommand({
Bucket: state.lockEnabledBucketName,
Key: "file1.txt",
Body: "Content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
await client.send(
new PutObjectCommand({
Bucket: state.retentionBucketName,
Key: "file0.txt",
Body: "Content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
await client.send(
new PutObjectCommand({
Bucket: state.retentionBucketName,
Key: "file1.txt",
Body: "Content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
} catch (caught) {
if (caught instanceof S3ServiceException) {
console.error(
`Error from S3 while uploading object. ${caught.name}: ${caught.message}`,
);
} else {
throw caught;
}
}
});
/**
* @param {Scenarios} scenarios
*/
const updateRetention = (scenarios) =>
new scenarios.ScenarioOutput(
"updateRetention",
(state) => `A bucket can be configured to use object locking with a default retention period.
A default retention period will be configured for ${state.bucketPrefix}-retention-after-creation.`,
{ preformatted: true },
);
/**
* @param {Scenarios} scenarios
*/
const confirmUpdateRetention = (scenarios) =>
new scenarios.ScenarioInput(
"confirmUpdateRetention",
"Configure default retention period?",
{ type: "confirm" },
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const updateRetentionAction = (scenarios, client) =>
new scenarios.ScenarioAction("updateRetentionAction", async (state) => {
await client.send(
new PutBucketVersioningCommand({
Bucket: state.retentionBucketName,
VersioningConfiguration: {
MFADelete: MFADeleteStatus.Disabled,
Status: BucketVersioningStatus.Enabled,
},
}),
);
const getBucketVersioning = new GetBucketVersioningCommand({
Bucket: state.retentionBucketName,
});
await retry({ intervalInMs: 500, maxRetries: 10 }, async () => {
const { Status } = await client.send(getBucketVersioning);
if (Status !== "Enabled") {
throw new Error("Bucket versioning is not enabled.");
}
});
await client.send(
new PutObjectLockConfigurationCommand({
Bucket: state.retentionBucketName,
ObjectLockConfiguration: {
ObjectLockEnabled: "Enabled",
Rule: {
DefaultRetention: {
Mode: "GOVERNANCE",
Years: 1,
},
},
},
}),
);
});
/**
* @param {Scenarios} scenarios
*/
const updateLockPolicy = (scenarios) =>
new scenarios.ScenarioOutput(
"updateLockPolicy",
(state) => `Object lock policies can also be added to existing buckets.
An object lock policy will be added to ${state.bucketPrefix}-lock-enabled.`,
{ preformatted: true },
);
/**
* @param {Scenarios} scenarios
*/
const confirmUpdateLockPolicy = (scenarios) =>
new scenarios.ScenarioInput(
"confirmUpdateLockPolicy",
"Add object lock policy?",
{ type: "confirm" },
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const updateLockPolicyAction = (scenarios, client) =>
new scenarios.ScenarioAction("updateLockPolicyAction", async (state) => {
await client.send(
new PutObjectLockConfigurationCommand({
Bucket: state.lockEnabledBucketName,
ObjectLockConfiguration: {
ObjectLockEnabled: "Enabled",
},
}),
);
});
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const confirmSetLegalHoldFileEnabled = (scenarios) =>
new scenarios.ScenarioInput(
"confirmSetLegalHoldFileEnabled",
(state) =>
`Would you like to add a legal hold to file0.txt in ${state.lockEnabledBucketName}?`,
{
type: "confirm",
},
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const setLegalHoldFileEnabledAction = (scenarios, client) =>
new scenarios.ScenarioAction(
"setLegalHoldFileEnabledAction",
async (state) => {
await client.send(
new PutObjectLegalHoldCommand({
Bucket: state.lockEnabledBucketName,
Key: "file0.txt",
LegalHold: {
Status: ObjectLockLegalHoldStatus.ON,
},
}),
);
console.log(
`Modified legal hold for file0.txt in ${state.lockEnabledBucketName}.`,
);
},
{ skipWhen: (state) => !state.confirmSetLegalHoldFileEnabled },
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const confirmSetRetentionPeriodFileEnabled = (scenarios) =>
new scenarios.ScenarioInput(
"confirmSetRetentionPeriodFileEnabled",
(state) =>
`Would you like to add a 1 day Governance retention period to file1.txt in ${state.lockEnabledBucketName}?
Reminder: Only a user with the s3:BypassGovernanceRetention permission will be able to delete this file or its bucket until the retention period has expired.`,
{
type: "confirm",
},
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const setRetentionPeriodFileEnabledAction = (scenarios, client) =>
new scenarios.ScenarioAction(
"setRetentionPeriodFileEnabledAction",
async (state) => {
const retentionDate = new Date();
retentionDate.setDate(retentionDate.getDate() + 1);
await client.send(
new PutObjectRetentionCommand({
Bucket: state.lockEnabledBucketName,
Key: "file1.txt",
Retention: {
Mode: ObjectLockRetentionMode.GOVERNANCE,
RetainUntilDate: retentionDate,
},
}),
);
console.log(
`Set retention for file1.txt in ${state.lockEnabledBucketName} until ${retentionDate.toISOString().split("T")[0]}.`,
);
},
{ skipWhen: (state) => !state.confirmSetRetentionPeriodFileEnabled },
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const confirmSetLegalHoldFileRetention = (scenarios) =>
new scenarios.ScenarioInput(
"confirmSetLegalHoldFileRetention",
(state) =>
`Would you like to add a legal hold to file0.txt in ${state.retentionBucketName}?`,
{
type: "confirm",
},
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const setLegalHoldFileRetentionAction = (scenarios, client) =>
new scenarios.ScenarioAction(
"setLegalHoldFileRetentionAction",
async (state) => {
await client.send(
new PutObjectLegalHoldCommand({
Bucket: state.retentionBucketName,
Key: "file0.txt",
LegalHold: {
Status: ObjectLockLegalHoldStatus.ON,
},
}),
);
console.log(
`Modified legal hold for file0.txt in ${state.retentionBucketName}.`,
);
},
{ skipWhen: (state) => !state.confirmSetLegalHoldFileRetention },
);
/**
* @param {Scenarios} scenarios
*/
const confirmSetRetentionPeriodFileRetention = (scenarios) =>
new scenarios.ScenarioInput(
"confirmSetRetentionPeriodFileRetention",
(state) =>
`Would you like to add a 1 day Governance retention period to file1.txt in ${state.retentionBucketName}?
Reminder: Only a user with the s3:BypassGovernanceRetention permission will be able to delete this file or its bucket until the retention period has expired.`,
{
type: "confirm",
},
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const setRetentionPeriodFileRetentionAction = (scenarios, client) =>
new scenarios.ScenarioAction(
"setRetentionPeriodFileRetentionAction",
async (state) => {
const retentionDate = new Date();
retentionDate.setDate(retentionDate.getDate() + 1);
await client.send(
new PutObjectRetentionCommand({
Bucket: state.retentionBucketName,
Key: "file1.txt",
Retention: {
Mode: ObjectLockRetentionMode.GOVERNANCE,
RetainUntilDate: retentionDate,
},
BypassGovernanceRetention: true,
}),
);
console.log(
`Set retention for file1.txt in ${state.retentionBucketName} until ${retentionDate.toISOString().split("T")[0]}.`,
);
},
{ skipWhen: (state) => !state.confirmSetRetentionPeriodFileRetention },
);
export {
getBucketPrefix,
createBuckets,
confirmCreateBuckets,
createBucketsAction,
populateBuckets,
confirmPopulateBuckets,
populateBucketsAction,
updateRetention,
confirmUpdateRetention,
updateRetentionAction,
updateLockPolicy,
confirmUpdateLockPolicy,
updateLockPolicyAction,
confirmSetLegalHoldFileEnabled,
setLegalHoldFileEnabledAction,
confirmSetRetentionPeriodFileEnabled,
setRetentionPeriodFileEnabledAction,
confirmSetLegalHoldFileRetention,
setLegalHoldFileRetentionAction,
confirmSetRetentionPeriodFileRetention,
setRetentionPeriodFileRetentionAction,
};
```
View and delete files in the buckets (repl.steps.js).
```
import {
ChecksumAlgorithm,
DeleteObjectCommand,
GetObjectLegalHoldCommand,
GetObjectLockConfigurationCommand,
GetObjectRetentionCommand,
ListObjectVersionsCommand,
PutObjectCommand,
} from "@aws-sdk/client-s3";
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @typedef {import("@aws-sdk/client-s3").S3Client} S3Client
*/
const choices = {
EXIT: 0,
LIST_ALL_FILES: 1,
DELETE_FILE: 2,
DELETE_FILE_WITH_RETENTION: 3,
OVERWRITE_FILE: 4,
VIEW_RETENTION_SETTINGS: 5,
VIEW_LEGAL_HOLD_SETTINGS: 6,
};
/**
* @param {Scenarios} scenarios
*/
const replInput = (scenarios) =>
new scenarios.ScenarioInput(
"replChoice",
"Explore the S3 locking features by selecting one of the following choices",
{
type: "select",
choices: [
{ name: "List all files in buckets", value: choices.LIST_ALL_FILES },
{ name: "Attempt to delete a file.", value: choices.DELETE_FILE },
{
name: "Attempt to delete a file with retention period bypass.",
value: choices.DELETE_FILE_WITH_RETENTION,
},
{ name: "Attempt to overwrite a file.", value: choices.OVERWRITE_FILE },
{
name: "View the object and bucket retention settings for a file.",
value: choices.VIEW_RETENTION_SETTINGS,
},
{
name: "View the legal hold settings for a file.",
value: choices.VIEW_LEGAL_HOLD_SETTINGS,
},
{ name: "Finish the workflow.", value: choices.EXIT },
],
},
);
/**
* @param {S3Client} client
* @param {string[]} buckets
*/
const getAllFiles = async (client, buckets) => {
/** @type {{bucket: string, key: string, version: string}[]} */
const files = [];
for (const bucket of buckets) {
const objectsResponse = await client.send(
new ListObjectVersionsCommand({ Bucket: bucket }),
);
for (const version of objectsResponse.Versions || []) {
const { Key, VersionId } = version;
files.push({ bucket, key: Key, version: VersionId });
}
}
return files;
};
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const replAction = (scenarios, client) =>
new scenarios.ScenarioAction(
"replAction",
async (state) => {
const files = await getAllFiles(client, [
state.noLockBucketName,
state.lockEnabledBucketName,
state.retentionBucketName,
]);
const fileInput = new scenarios.ScenarioInput(
"selectedFile",
"Select a file:",
{
type: "select",
choices: files.map((file, index) => ({
name: `${index + 1}: ${file.bucket}: ${file.key} (version: ${
file.version
})`,
value: index,
})),
},
);
const { replChoice } = state;
switch (replChoice) {
case choices.LIST_ALL_FILES: {
const files = await getAllFiles(client, [
state.noLockBucketName,
state.lockEnabledBucketName,
state.retentionBucketName,
]);
state.replOutput = files
.map(
(file) =>
`${file.bucket}: ${file.key} (version: ${file.version})`,
)
.join("\n");
break;
}
case choices.DELETE_FILE: {
/** @type {number} */
const fileToDelete = await fileInput.handle(state);
const selectedFile = files[fileToDelete];
try {
await client.send(
new DeleteObjectCommand({
Bucket: selectedFile.bucket,
Key: selectedFile.key,
VersionId: selectedFile.version,
}),
);
state.replOutput = `Deleted ${selectedFile.key} in ${selectedFile.bucket}.`;
} catch (err) {
state.replOutput = `Unable to delete object ${selectedFile.key} in bucket ${selectedFile.bucket}: ${err.message}`;
}
break;
}
case choices.DELETE_FILE_WITH_RETENTION: {
/** @type {number} */
const fileToDelete = await fileInput.handle(state);
const selectedFile = files[fileToDelete];
try {
await client.send(
new DeleteObjectCommand({
Bucket: selectedFile.bucket,
Key: selectedFile.key,
VersionId: selectedFile.version,
BypassGovernanceRetention: true,
}),
);
state.replOutput = `Deleted ${selectedFile.key} in ${selectedFile.bucket}.`;
} catch (err) {
state.replOutput = `Unable to delete object ${selectedFile.key} in bucket ${selectedFile.bucket}: ${err.message}`;
}
break;
}
case choices.OVERWRITE_FILE: {
/** @type {number} */
const fileToOverwrite = await fileInput.handle(state);
const selectedFile = files[fileToOverwrite];
try {
await client.send(
new PutObjectCommand({
Bucket: selectedFile.bucket,
Key: selectedFile.key,
Body: "New content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
state.replOutput = `Overwrote ${selectedFile.key} in ${selectedFile.bucket}.`;
} catch (err) {
state.replOutput = `Unable to overwrite object ${selectedFile.key} in bucket ${selectedFile.bucket}: ${err.message}`;
}
break;
}
case choices.VIEW_RETENTION_SETTINGS: {
/** @type {number} */
const fileToView = await fileInput.handle(state);
const selectedFile = files[fileToView];
try {
const retention = await client.send(
new GetObjectRetentionCommand({
Bucket: selectedFile.bucket,
Key: selectedFile.key,
VersionId: selectedFile.version,
}),
);
const bucketConfig = await client.send(
new GetObjectLockConfigurationCommand({
Bucket: selectedFile.bucket,
}),
);
state.replOutput = `Object retention for ${selectedFile.key} in ${selectedFile.bucket}: ${retention.Retention?.Mode} until ${retention.Retention?.RetainUntilDate?.toISOString()}.
Bucket object lock config for ${selectedFile.bucket} in ${selectedFile.bucket}:
Enabled: ${bucketConfig.ObjectLockConfiguration?.ObjectLockEnabled}
Rule: ${JSON.stringify(bucketConfig.ObjectLockConfiguration?.Rule?.DefaultRetention)}`;
} catch (err) {
state.replOutput = `Unable to fetch object lock retention: '${err.message}'`;
}
break;
}
case choices.VIEW_LEGAL_HOLD_SETTINGS: {
/** @type {number} */
const fileToView = await fileInput.handle(state);
const selectedFile = files[fileToView];
try {
const legalHold = await client.send(
new GetObjectLegalHoldCommand({
Bucket: selectedFile.bucket,
Key: selectedFile.key,
VersionId: selectedFile.version,
}),
);
state.replOutput = `Object legal hold for ${selectedFile.key} in ${selectedFile.bucket}: Status: ${legalHold.LegalHold?.Status}`;
} catch (err) {
state.replOutput = `Unable to fetch legal hold: '${err.message}'`;
}
break;
}
default:
throw new Error(`Invalid replChoice: ${replChoice}`);
}
},
{
whileConfig: {
whileFn: ({ replChoice }) => replChoice !== choices.EXIT,
input: replInput(scenarios),
output: new scenarios.ScenarioOutput(
"REPL output",
(state) => state.replOutput,
{ preformatted: true },
),
},
},
);
export { replInput, replAction, choices };
```
Destroy all created resources (clean.steps.js).
```
import {
DeleteObjectCommand,
DeleteBucketCommand,
ListObjectVersionsCommand,
GetObjectLegalHoldCommand,
GetObjectRetentionCommand,
PutObjectLegalHoldCommand,
} from "@aws-sdk/client-s3";
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @typedef {import("@aws-sdk/client-s3").S3Client} S3Client
*/
/**
* @param {Scenarios} scenarios
*/
const confirmCleanup = (scenarios) =>
new scenarios.ScenarioInput("confirmCleanup", "Clean up resources?", {
type: "confirm",
});
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const cleanupAction = (scenarios, client) =>
new scenarios.ScenarioAction("cleanupAction", async (state) => {
const { noLockBucketName, lockEnabledBucketName, retentionBucketName } =
state;
const buckets = [
noLockBucketName,
lockEnabledBucketName,
retentionBucketName,
];
for (const bucket of buckets) {
/** @type {import("@aws-sdk/client-s3").ListObjectVersionsCommandOutput} */
let objectsResponse;
try {
objectsResponse = await client.send(
new ListObjectVersionsCommand({
Bucket: bucket,
}),
);
} catch (e) {
if (e instanceof Error && e.name === "NoSuchBucket") {
console.log("Object's bucket has already been deleted.");
continue;
}
throw e;
}
for (const version of objectsResponse.Versions || []) {
const { Key, VersionId } = version;
try {
const legalHold = await client.send(
new GetObjectLegalHoldCommand({
Bucket: bucket,
Key,
VersionId,
}),
);
if (legalHold.LegalHold?.Status === "ON") {
await client.send(
new PutObjectLegalHoldCommand({
Bucket: bucket,
Key,
VersionId,
LegalHold: {
Status: "OFF",
},
}),
);
}
} catch (err) {
console.log(
`Unable to fetch legal hold for ${Key} in ${bucket}: '${err.message}'`,
);
}
try {
const retention = await client.send(
new GetObjectRetentionCommand({
Bucket: bucket,
Key,
VersionId,
}),
);
if (retention.Retention?.Mode === "GOVERNANCE") {
await client.send(
new DeleteObjectCommand({
Bucket: bucket,
Key,
VersionId,
BypassGovernanceRetention: true,
}),
);
}
} catch (err) {
console.log(
`Unable to fetch object lock retention for ${Key} in ${bucket}: '${err.message}'`,
);
}
await client.send(
new DeleteObjectCommand({
Bucket: bucket,
Key,
VersionId,
}),
);
}
await client.send(new DeleteBucketCommand({ Bucket: bucket }));
console.log(`Delete for ${bucket} complete.`);
}
});
export { confirmCleanup, cleanupAction };
```
+ For API details, see the following topics in *AWS SDK for JavaScript API Reference*.
+ [GetObjectLegalHold](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/GetObjectLegalHoldCommand)
+ [GetObjectLockConfiguration](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/GetObjectLockConfigurationCommand)
+ [GetObjectRetention](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/GetObjectRetentionCommand)
+ [PutObjectLegalHold](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/PutObjectLegalHoldCommand)
+ [PutObjectLockConfiguration](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/PutObjectLockConfigurationCommand)
+ [PutObjectRetention](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/PutObjectRetentionCommand)
------
# Make Amazon S3 conditional requests using an AWS SDK
The following code examples show how to add preconditions to Amazon S3 requests.
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/scenarios/S3ConditionalRequestsScenario#code-examples).
Run an interactive scenario demonstrating Amazon S3 conditional request features.
```
using Amazon.S3;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Console;
using Microsoft.Extensions.Logging.Debug;
namespace S3ConditionalRequestsScenario;
public static class S3ConditionalRequestsScenario
{
/*
Before running this .NET code example, set up your development environment, including your credentials.
This example demonstrates the use of conditional requests for S3 operations.
You can use conditional requests to add preconditions to S3 read requests to return or copy
an object based on its Entity tag (ETag), or last modified date.
You can use a conditional write requests to prevent overwrites by ensuring
there is no existing object with the same key.
*/
public static S3ActionsWrapper _s3ActionsWrapper = null!;
public static IConfiguration _configuration = null!;
public static string _resourcePrefix = null!;
public static string _sourceBucketName = null!;
public static string _destinationBucketName = null!;
public static string _sampleObjectKey = null!;
public static string _sampleObjectEtag = null!;
public static bool _interactive = true;
public static async Task Main(string[] args)
{
// Set up dependency injection for the Amazon service.
using var host = Host.CreateDefaultBuilder(args)
.ConfigureLogging(logging =>
logging.AddFilter("System", LogLevel.Debug)
.AddFilter("Microsoft", LogLevel.Information)
.AddFilter("Microsoft", LogLevel.Trace))
.ConfigureServices((_, services) =>
services.AddAWSService()
.AddTransient()
)
.Build();
_configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("settings.json") // Load settings from .json file.
.AddJsonFile("settings.local.json",
true) // Optionally, load local settings.
.Build();
ServicesSetup(host);
try
{
Console.WriteLine(new string('-', 80));
Console.WriteLine("Welcome to the Amazon Simple Storage Service (S3) Conditional Requests Feature Scenario.");
Console.WriteLine(new string('-', 80));
ConfigurationSetup();
_sampleObjectEtag = await Setup(_sourceBucketName, _destinationBucketName, _sampleObjectKey);
await DisplayDemoChoices(_sourceBucketName, _destinationBucketName, _sampleObjectKey, _sampleObjectEtag, 0);
Console.WriteLine(new string('-', 80));
Console.WriteLine("Cleaning up resources.");
Console.WriteLine(new string('-', 80));
await Cleanup(true);
Console.WriteLine(new string('-', 80));
Console.WriteLine("Amazon S3 Conditional Requests Feature Scenario is complete.");
Console.WriteLine(new string('-', 80));
}
catch (Exception ex)
{
Console.WriteLine(new string('-', 80));
Console.WriteLine($"There was a problem: {ex.Message}");
await CleanupScenario(_sourceBucketName, _destinationBucketName);
Console.WriteLine(new string('-', 80));
}
}
///
/// Populate the services for use within the console application.
///
/// The services host.
private static void ServicesSetup(IHost host)
{
_s3ActionsWrapper = host.Services.GetRequiredService();
}
///
/// Any setup operations needed.
///
public static void ConfigurationSetup()
{
_resourcePrefix = _configuration["resourcePrefix"] ?? "dotnet-example";
_sourceBucketName = _resourcePrefix + "-source";
_destinationBucketName = _resourcePrefix + "-dest";
_sampleObjectKey = _resourcePrefix + "-sample-object.txt";
}
///
/// Sets up the scenario by creating a source and destination bucket, and uploading a test file to the source bucket.
///
/// The name of the source bucket.
/// The name of the destination bucket.
/// The name of the test file to add to the source bucket.
/// The ETag of the uploaded test file.
public static async Task Setup(string sourceBucket, string destBucket, string objectKey)
{
Console.WriteLine(
"\nFor this scenario, we will use the AWS SDK for .NET to create several S3\n" +
"buckets and files to demonstrate working with S3 conditional requests.\n" +
"This example demonstrates the use of conditional requests for S3 operations.\r\n" +
"You can use conditional requests to add preconditions to S3 read requests to return or copy\r\n" +
"an object based on its Entity tag (ETag), or last modified date. \r\n" +
"You can use a conditional write requests to prevent overwrites by ensuring \r\n" +
"there is no existing object with the same key. \r\n\r\n" +
"This example will allow you to perform conditional reads\r\n" +
"and writes that will succeed or fail based on your selected options.\r\n\r\n" +
"Sample buckets and a sample object will be created as part of the example.");
Console.WriteLine(new string('-', 80));
Console.WriteLine("Press Enter when you are ready to start.");
if (_interactive)
Console.ReadLine();
await _s3ActionsWrapper.CreateBucketWithName(sourceBucket);
await _s3ActionsWrapper.CreateBucketWithName(destBucket);
var eTag = await _s3ActionsWrapper.PutObjectConditional(objectKey, sourceBucket,
"Test file content.");
return eTag;
}
///
/// Cleans up the scenario by deleting the source and destination buckets.
///
/// The name of the source bucket.
/// The name of the destination bucket.
public static async Task CleanupScenario(string sourceBucket, string destBucket)
{
await _s3ActionsWrapper.CleanupBucketByName(sourceBucket);
await _s3ActionsWrapper.CleanupBucketByName(destBucket);
}
///
/// Displays a list of the objects in the test buckets.
///
/// The name of the source bucket.
/// The name of the destination bucket.
public static async Task DisplayBuckets(string sourceBucket, string destBucket)
{
await _s3ActionsWrapper.ListBucketContentsByName(sourceBucket);
await _s3ActionsWrapper.ListBucketContentsByName(destBucket);
}
///
/// Displays the menu of conditional request options for the user.
///
/// The name of the source bucket.
/// The name of the destination bucket.
/// The key of the test object in the source bucket.
/// The ETag of the test object in the source bucket.
public static async Task DisplayDemoChoices(string sourceBucket, string destBucket, string objectKey, string etag, int defaultChoice)
{
var actions = new[]
{
"Print a list of bucket items.",
"Perform a conditional read.",
"Perform a conditional copy.",
"Perform a conditional write.",
"Clean up and exit."
};
var conditions = new[]
{
"If-Match: using the object's ETag. This condition should succeed.",
"If-None-Match: using the object's ETag. This condition should fail.",
"If-Modified-Since: using yesterday's date. This condition should succeed.",
"If-Unmodified-Since: using yesterday's date. This condition should fail."
};
var conditionTypes = new[]
{
S3ConditionType.IfMatch,
S3ConditionType.IfNoneMatch,
S3ConditionType.IfModifiedSince,
S3ConditionType.IfUnmodifiedSince,
};
var yesterdayDate = DateTime.UtcNow.AddDays(-1);
int choice;
while ((choice = GetChoiceResponse("\nExplore the S3 conditional request features by selecting one of the following choices:", actions, defaultChoice)) != 4)
{
switch (choice)
{
case 0:
Console.WriteLine("Listing the objects and buckets.");
await DisplayBuckets(sourceBucket, destBucket);
break;
case 1:
int conditionTypeIndex = GetChoiceResponse("Perform a conditional read:", conditions, 1);
if (conditionTypeIndex == 0 || conditionTypeIndex == 1)
{
await _s3ActionsWrapper.GetObjectConditional(objectKey, sourceBucket, conditionTypes[conditionTypeIndex], null, _sampleObjectEtag);
}
else if (conditionTypeIndex == 2 || conditionTypeIndex == 3)
{
await _s3ActionsWrapper.GetObjectConditional(objectKey, sourceBucket, conditionTypes[conditionTypeIndex], yesterdayDate);
}
break;
case 2:
int copyConditionTypeIndex = GetChoiceResponse("Perform a conditional copy:", conditions, 1);
string destKey = GetStringResponse("Enter an object key:", "sampleObjectKey");
if (copyConditionTypeIndex == 0 || copyConditionTypeIndex == 1)
{
await _s3ActionsWrapper.CopyObjectConditional(objectKey, destKey, sourceBucket, destBucket, conditionTypes[copyConditionTypeIndex], null, etag);
}
else if (copyConditionTypeIndex == 2 || copyConditionTypeIndex == 3)
{
await _s3ActionsWrapper.CopyObjectConditional(objectKey, destKey, sourceBucket, destBucket, conditionTypes[copyConditionTypeIndex], yesterdayDate);
}
break;
case 3:
Console.WriteLine("Perform a conditional write using IfNoneMatch condition on the object key.");
Console.WriteLine("If the key is a duplicate, the write will fail.");
string newObjectKey = GetStringResponse("Enter an object key:", "newObjectKey");
await _s3ActionsWrapper.PutObjectConditional(newObjectKey, sourceBucket, "Conditional write example data.");
break;
}
if (!_interactive)
{
break;
}
}
Console.WriteLine("Proceeding to cleanup.");
}
//
/// Clean up the resources from the scenario.
///
/// True to run as interactive.
/// True if successful.
public static async Task Cleanup(bool interactive)
{
Console.WriteLine(new string('-', 80));
if (!interactive || GetYesNoResponse("Do you want to clean up all files and buckets? (y/n) "))
{
await _s3ActionsWrapper.CleanUpBucketByName(_sourceBucketName);
await _s3ActionsWrapper.CleanUpBucketByName(_destinationBucketName);
}
else
{
Console.WriteLine(
"Ok, we'll leave the resources intact.\n" +
"Don't forget to delete them when you're done with them or you might incur unexpected charges."
);
}
Console.WriteLine(new string('-', 80));
return true;
}
///
/// Helper method to get a yes or no response from the user.
///
/// The question string to print on the console.
/// True if the user responds with a yes.
private static bool GetYesNoResponse(string question)
{
Console.WriteLine(question);
var ynResponse = Console.ReadLine();
var response = ynResponse != null && ynResponse.Equals("y", StringComparison.InvariantCultureIgnoreCase);
return response;
}
///
/// Helper method to get a choice response from the user.
///
/// The question string to print on the console.
/// The choices to print on the console.
/// The index of the selected choice
private static int GetChoiceResponse(string? question, string[] choices, int defaultChoice)
{
if (question != null)
{
Console.WriteLine(question);
for (int i = 0; i < choices.Length; i++)
{
Console.WriteLine($"\t{i + 1}. {choices[i]}");
}
}
if (!_interactive)
return defaultChoice;
var choiceNumber = 0;
while (choiceNumber < 1 || choiceNumber > choices.Length)
{
var choice = Console.ReadLine();
Int32.TryParse(choice, out choiceNumber);
}
return choiceNumber - 1;
}
///
/// Get a string response from the user.
///
/// The question to print.
/// A default answer to use when not interactive.
/// The string response.
public static string GetStringResponse(string? question, string defaultAnswer)
{
string? answer = "";
if (_interactive)
{
do
{
Console.WriteLine(question);
answer = Console.ReadLine();
} while (string.IsNullOrWhiteSpace(answer));
}
else
{
answer = defaultAnswer;
}
return answer;
}
}
```
A wrapper class for S3 functions.
```
using System.Net;
using Amazon.S3;
using Amazon.S3.Model;
using Microsoft.Extensions.Logging;
namespace S3ConditionalRequestsScenario;
///
/// Encapsulate the Amazon S3 operations.
///
public class S3ActionsWrapper
{
private readonly IAmazonS3 _amazonS3;
private readonly ILogger _logger;
///
/// Constructor for the S3ActionsWrapper.
///
/// The injected S3 client.
/// The class logger.
public S3ActionsWrapper(IAmazonS3 amazonS3, ILogger logger)
{
_amazonS3 = amazonS3;
_logger = logger;
}
///
/// Retrieves an object from Amazon S3 with a conditional request.
///
/// The key of the object to retrieve.
/// The source bucket of the object.
/// The type of condition: 'IfMatch', 'IfNoneMatch', 'IfModifiedSince', 'IfUnmodifiedSince'.
/// The value to use for the condition for dates.
/// The value to use for the condition for etags.
/// True if the conditional read is successful, False otherwise.
public async Task GetObjectConditional(string objectKey, string sourceBucket,
S3ConditionType conditionType, DateTime? conditionDateValue = null, string? etagConditionalValue = null)
{
try
{
var getObjectRequest = new GetObjectRequest
{
BucketName = sourceBucket,
Key = objectKey
};
switch (conditionType)
{
case S3ConditionType.IfMatch:
getObjectRequest.EtagToMatch = etagConditionalValue;
break;
case S3ConditionType.IfNoneMatch:
getObjectRequest.EtagToNotMatch = etagConditionalValue;
break;
case S3ConditionType.IfModifiedSince:
getObjectRequest.ModifiedSinceDateUtc = conditionDateValue.GetValueOrDefault();
break;
case S3ConditionType.IfUnmodifiedSince:
getObjectRequest.UnmodifiedSinceDateUtc = conditionDateValue.GetValueOrDefault();
break;
default:
throw new ArgumentOutOfRangeException(nameof(conditionType), conditionType, null);
}
var response = await _amazonS3.GetObjectAsync(getObjectRequest);
var sampleBytes = new byte[20];
await response.ResponseStream.ReadAsync(sampleBytes, 0, 20);
_logger.LogInformation($"Conditional read successful. Here are the first 20 bytes of the object:\n{System.Text.Encoding.UTF8.GetString(sampleBytes)}");
return true;
}
catch (AmazonS3Exception e)
{
if (e.ErrorCode == "PreconditionFailed")
{
_logger.LogError("Conditional read failed: Precondition failed");
}
else if (e.ErrorCode == "NotModified")
{
_logger.LogError("Conditional read failed: Object not modified");
}
else
{
_logger.LogError($"Unexpected error: {e.ErrorCode}");
throw;
}
return false;
}
}
///
/// Uploads an object to Amazon S3 with a conditional request. Prevents overwrite using an IfNoneMatch condition for the object key.
///
/// The key of the object to upload.
/// The source bucket of the object.
/// The content to upload as a string.
/// The ETag if the conditional write is successful, empty otherwise.
public async Task PutObjectConditional(string objectKey, string bucket, string content)
{
try
{
var putObjectRequest = new PutObjectRequest
{
BucketName = bucket,
Key = objectKey,
ContentBody = content,
IfNoneMatch = "*"
};
var putResult = await _amazonS3.PutObjectAsync(putObjectRequest);
_logger.LogInformation($"Conditional write successful for key {objectKey} in bucket {bucket}.");
return putResult.ETag;
}
catch (AmazonS3Exception e)
{
if (e.ErrorCode == "PreconditionFailed")
{
_logger.LogError("Conditional write failed: Precondition failed");
}
else
{
_logger.LogError($"Unexpected error: {e.ErrorCode}");
throw;
}
return string.Empty;
}
}
///
/// Copies an object from one Amazon S3 bucket to another with a conditional request.
///
/// The key of the source object to copy.
/// The key of the destination object.
/// The source bucket of the object.
/// The destination bucket of the object.
/// The type of condition to apply, e.g. 'CopySourceIfMatch', 'CopySourceIfNoneMatch', 'CopySourceIfModifiedSince', 'CopySourceIfUnmodifiedSince'.
/// The value to use for the condition for dates.
/// The value to use for the condition for etags.
/// True if the conditional copy is successful, False otherwise.
public async Task CopyObjectConditional(string sourceKey, string destKey, string sourceBucket, string destBucket,
S3ConditionType conditionType, DateTime? conditionDateValue = null, string? etagConditionalValue = null)
{
try
{
var copyObjectRequest = new CopyObjectRequest
{
DestinationBucket = destBucket,
DestinationKey = destKey,
SourceBucket = sourceBucket,
SourceKey = sourceKey
};
switch (conditionType)
{
case S3ConditionType.IfMatch:
copyObjectRequest.ETagToMatch = etagConditionalValue;
break;
case S3ConditionType.IfNoneMatch:
copyObjectRequest.ETagToNotMatch = etagConditionalValue;
break;
case S3ConditionType.IfModifiedSince:
copyObjectRequest.ModifiedSinceDateUtc = conditionDateValue.GetValueOrDefault();
break;
case S3ConditionType.IfUnmodifiedSince:
copyObjectRequest.UnmodifiedSinceDateUtc = conditionDateValue.GetValueOrDefault();
break;
default:
throw new ArgumentOutOfRangeException(nameof(conditionType), conditionType, null);
}
await _amazonS3.CopyObjectAsync(copyObjectRequest);
_logger.LogInformation($"Conditional copy successful for key {destKey} in bucket {destBucket}.");
return true;
}
catch (AmazonS3Exception e)
{
if (e.ErrorCode == "PreconditionFailed")
{
_logger.LogError("Conditional copy failed: Precondition failed");
}
else if (e.ErrorCode == "304")
{
_logger.LogError("Conditional copy failed: Object not modified");
}
else
{
_logger.LogError($"Unexpected error: {e.ErrorCode}");
throw;
}
return false;
}
}
///
/// Create a new Amazon S3 bucket with a specified name and check that the bucket is ready.
///
/// The name of the bucket to create.
/// True if successful.
public async Task CreateBucketWithName(string bucketName)
{
Console.WriteLine($"\tCreating bucket {bucketName}.");
try
{
var request = new PutBucketRequest
{
BucketName = bucketName,
UseClientRegion = true
};
await _amazonS3.PutBucketAsync(request);
var bucketReady = false;
var retries = 5;
while (!bucketReady && retries > 0)
{
Thread.Sleep(5000);
bucketReady = await Amazon.S3.Util.AmazonS3Util.DoesS3BucketExistV2Async(_amazonS3, bucketName);
retries--;
}
return bucketReady;
}
catch (BucketAlreadyExistsException ex)
{
Console.WriteLine($"Bucket already exists: '{ex.Message}'");
return true;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"Error creating bucket: '{ex.Message}'");
return false;
}
}
///
/// Cleans up objects and deletes the bucket by name.
///
/// The name of the bucket.
/// Async task.
public async Task CleanupBucketByName(string bucketName)
{
try
{
var listObjectsResponse = await _amazonS3.ListObjectsV2Async(new ListObjectsV2Request { BucketName = bucketName });
foreach (var obj in listObjectsResponse.S3Objects)
{
await _amazonS3.DeleteObjectAsync(new DeleteObjectRequest { BucketName = bucketName, Key = obj.Key });
}
await _amazonS3.DeleteBucketAsync(new DeleteBucketRequest { BucketName = bucketName });
Console.WriteLine($"Cleaned up bucket: {bucketName}.");
}
catch (AmazonS3Exception e)
{
if (e.ErrorCode == "NoSuchBucket")
{
Console.WriteLine($"Bucket {bucketName} does not exist, skipping cleanup.");
}
else
{
Console.WriteLine($"Error deleting bucket: {e.ErrorCode}");
throw;
}
}
}
///
/// List the contents of the bucket with their ETag.
///
/// The name of the bucket.
/// Async task.
public async Task> ListBucketContentsByName(string bucketName)
{
var results = new List();
try
{
Console.WriteLine($"\t Items in bucket {bucketName}");
var listObjectsResponse = await _amazonS3.ListObjectsV2Async(new ListObjectsV2Request { BucketName = bucketName });
if (listObjectsResponse.S3Objects.Count == 0)
{
Console.WriteLine("\t\tNo objects found.");
}
else
{
foreach (var obj in listObjectsResponse.S3Objects)
{
Console.WriteLine($"\t\t object: {obj.Key} ETag {obj.ETag}");
}
}
results = listObjectsResponse.S3Objects;
}
catch (AmazonS3Exception e)
{
if (e.ErrorCode == "NoSuchBucket")
{
_logger.LogError($"Bucket {bucketName} does not exist.");
}
else
{
_logger.LogError($"Error listing bucket and objects: {e.ErrorCode}");
throw;
}
}
return results;
}
///
/// Delete an object from a specific bucket.
///
/// The Amazon S3 bucket to use.
/// The key of the object to delete.
/// True if successful.
public async Task DeleteObjectFromBucket(string bucketName, string objectKey)
{
try
{
var request = new DeleteObjectRequest()
{
BucketName = bucketName,
Key = objectKey
};
await _amazonS3.DeleteObjectAsync(request);
Console.WriteLine($"Deleted {objectKey} in {bucketName}.");
return true;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tUnable to delete object {objectKey} in bucket {bucketName}: " + ex.Message);
return false;
}
}
///
/// Delete a specific bucket by deleting the objects and then the bucket itself.
///
/// The Amazon S3 bucket to use.
/// The key of the object to delete.
/// Optional versionId.
/// True if successful.
public async Task CleanUpBucketByName(string bucketName)
{
try
{
var allFiles = await ListBucketContentsByName(bucketName);
foreach (var fileInfo in allFiles)
{
await DeleteObjectFromBucket(fileInfo.BucketName, fileInfo.Key);
}
var request = new DeleteBucketRequest() { BucketName = bucketName, };
var response = await _amazonS3.DeleteBucketAsync(request);
Console.WriteLine($"\tDelete for {bucketName} complete.");
return response.HttpStatusCode == HttpStatusCode.OK;
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"\tUnable to delete bucket {bucketName}: " + ex.Message);
return false;
}
}
}
```
+ For API details, see the following topics in *AWS SDK for .NET API Reference*.
+ [CopyObject](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/CopyObject)
+ [GetObject](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObject)
+ [PutObject](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/PutObject)
------
#### [ JavaScript ]
**SDK for JavaScript (v3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javascriptv3/example_code/s3/scenarios/conditional-requests#code-examples).
Entrypoint for the workflow (index.js). This orchestrates all of the steps. Visit GitHub to see the implementation details for Scenario, ScenarioInput, ScenarioOutput, and ScenarioAction.
```
import * as Scenarios from "@aws-doc-sdk-examples/lib/scenario/index.js";
import {
exitOnFalse,
loadState,
saveState,
} from "@aws-doc-sdk-examples/lib/scenario/steps-common.js";
import { welcome, welcomeContinue } from "./welcome.steps.js";
import {
confirmCreateBuckets,
confirmPopulateBuckets,
createBuckets,
createBucketsAction,
getBucketPrefix,
populateBuckets,
populateBucketsAction,
} from "./setup.steps.js";
/**
* @param {Scenarios} scenarios
* @param {Record} initialState
*/
export const getWorkflowStages = (scenarios, initialState = {}) => {
const client = new S3Client({});
return {
deploy: new scenarios.Scenario(
"S3 Conditional Requests - Deploy",
[
welcome(scenarios),
welcomeContinue(scenarios),
exitOnFalse(scenarios, "welcomeContinue"),
getBucketPrefix(scenarios),
createBuckets(scenarios),
confirmCreateBuckets(scenarios),
exitOnFalse(scenarios, "confirmCreateBuckets"),
createBucketsAction(scenarios, client),
populateBuckets(scenarios),
confirmPopulateBuckets(scenarios),
exitOnFalse(scenarios, "confirmPopulateBuckets"),
populateBucketsAction(scenarios, client),
saveState,
],
initialState,
),
demo: new scenarios.Scenario(
"S3 Conditional Requests - Demo",
[loadState, welcome(scenarios), replAction(scenarios, client)],
initialState,
),
clean: new scenarios.Scenario(
"S3 Conditional Requests - Destroy",
[
loadState,
confirmCleanup(scenarios),
exitOnFalse(scenarios, "confirmCleanup"),
cleanupAction(scenarios, client),
],
initialState,
),
};
};
// Call function if run directly
import { fileURLToPath } from "node:url";
import { S3Client } from "@aws-sdk/client-s3";
import { cleanupAction, confirmCleanup } from "./clean.steps.js";
import { replAction } from "./repl.steps.js";
if (process.argv[1] === fileURLToPath(import.meta.url)) {
const objectLockingScenarios = getWorkflowStages(Scenarios);
Scenarios.parseScenarioArgs(objectLockingScenarios, {
name: "Amazon S3 object locking workflow",
description:
"Work with Amazon Simple Storage Service (Amazon S3) object locking features.",
synopsis:
"node index.js --scenario [-h|--help] [-y|--yes] [-v|--verbose]",
});
}
```
Output welcome messages to the console (welcome.steps.js).
```
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @param {Scenarios} scenarios
*/
const welcome = (scenarios) =>
new scenarios.ScenarioOutput(
"welcome",
"This example demonstrates the use of conditional requests for S3 operations." +
" You can use conditional requests to add preconditions to S3 read requests to return " +
"or copy an object based on its Entity tag (ETag), or last modified date.You can use " +
"a conditional write requests to prevent overwrites by ensuring there is no existing " +
"object with the same key.\n" +
"This example will enable you to perform conditional reads and writes that will succeed " +
"or fail based on your selected options.\n" +
"Sample buckets and a sample object will be created as part of the example.\n" +
"Some steps require a key name prefix to be defined by the user. Before you begin, you can " +
"optionally edit this prefix in ./object_name.json. If you do so, please reload the scenario before you begin.",
{ header: true },
);
/**
* @param {Scenarios} scenarios
*/
const welcomeContinue = (scenarios) =>
new scenarios.ScenarioInput(
"welcomeContinue",
"Press Enter when you are ready to start.",
{ type: "confirm" },
);
export { welcome, welcomeContinue };
```
Deploy buckets and objects (setup.steps.js).
```
import {
ChecksumAlgorithm,
CreateBucketCommand,
PutObjectCommand,
BucketAlreadyExists,
BucketAlreadyOwnedByYou,
S3ServiceException,
waitUntilBucketExists,
} from "@aws-sdk/client-s3";
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @typedef {import("@aws-sdk/client-s3").S3Client} S3Client
*/
/**
* @param {Scenarios} scenarios
*/
const getBucketPrefix = (scenarios) =>
new scenarios.ScenarioInput(
"bucketPrefix",
"Provide a prefix that will be used for bucket creation.",
{ type: "input", default: "amzn-s3-demo-bucket" },
);
/**
* @param {Scenarios} scenarios
*/
const createBuckets = (scenarios) =>
new scenarios.ScenarioOutput(
"createBuckets",
(state) => `The following buckets will be created:
${state.bucketPrefix}-source-bucket.
${state.bucketPrefix}-destination-bucket.`,
{ preformatted: true },
);
/**
* @param {Scenarios} scenarios
*/
const confirmCreateBuckets = (scenarios) =>
new scenarios.ScenarioInput("confirmCreateBuckets", "Create the buckets?", {
type: "confirm",
});
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const createBucketsAction = (scenarios, client) =>
new scenarios.ScenarioAction("createBucketsAction", async (state) => {
const sourceBucketName = `${state.bucketPrefix}-source-bucket`;
const destinationBucketName = `${state.bucketPrefix}-destination-bucket`;
try {
await client.send(
new CreateBucketCommand({
Bucket: sourceBucketName,
}),
);
await waitUntilBucketExists({ client }, { Bucket: sourceBucketName });
await client.send(
new CreateBucketCommand({
Bucket: destinationBucketName,
}),
);
await waitUntilBucketExists(
{ client },
{ Bucket: destinationBucketName },
);
state.sourceBucketName = sourceBucketName;
state.destinationBucketName = destinationBucketName;
} catch (caught) {
if (
caught instanceof BucketAlreadyExists ||
caught instanceof BucketAlreadyOwnedByYou
) {
console.error(`${caught.name}: ${caught.message}`);
state.earlyExit = true;
} else {
throw caught;
}
}
});
/**
* @param {Scenarios} scenarios
*/
const populateBuckets = (scenarios) =>
new scenarios.ScenarioOutput(
"populateBuckets",
(state) => `The following test files will be created:
file01.txt in ${state.bucketPrefix}-source-bucket.`,
{ preformatted: true },
);
/**
* @param {Scenarios} scenarios
*/
const confirmPopulateBuckets = (scenarios) =>
new scenarios.ScenarioInput(
"confirmPopulateBuckets",
"Populate the buckets?",
{ type: "confirm" },
);
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const populateBucketsAction = (scenarios, client) =>
new scenarios.ScenarioAction("populateBucketsAction", async (state) => {
try {
await client.send(
new PutObjectCommand({
Bucket: state.sourceBucketName,
Key: "file01.txt",
Body: "Content",
ChecksumAlgorithm: ChecksumAlgorithm.SHA256,
}),
);
} catch (caught) {
if (caught instanceof S3ServiceException) {
console.error(
`Error from S3 while uploading object. ${caught.name}: ${caught.message}`,
);
} else {
throw caught;
}
}
});
export {
confirmCreateBuckets,
confirmPopulateBuckets,
createBuckets,
createBucketsAction,
getBucketPrefix,
populateBuckets,
populateBucketsAction,
};
```
Get, copy, and put objects using S3 conditional requests (repl.steps.js).
```
import path from "node:path";
import { fileURLToPath } from "node:url";
import { dirname } from "node:path";
import {
ListObjectVersionsCommand,
GetObjectCommand,
CopyObjectCommand,
PutObjectCommand,
} from "@aws-sdk/client-s3";
import data from "./object_name.json" assert { type: "json" };
import { readFile } from "node:fs/promises";
import {
ScenarioInput,
Scenario,
ScenarioAction,
ScenarioOutput,
} from "../../../libs/scenario/index.js";
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @typedef {import("@aws-sdk/client-s3").S3Client} S3Client
*/
const choices = {
EXIT: 0,
LIST_ALL_FILES: 1,
CONDITIONAL_READ: 2,
CONDITIONAL_COPY: 3,
CONDITIONAL_WRITE: 4,
};
/**
* @param {Scenarios} scenarios
*/
const replInput = (scenarios) =>
new ScenarioInput(
"replChoice",
"Explore the S3 conditional request features by selecting one of the following choices",
{
type: "select",
choices: [
{ name: "Print list of bucket items.", value: choices.LIST_ALL_FILES },
{
name: "Perform a conditional read.",
value: choices.CONDITIONAL_READ,
},
{
name: "Perform a conditional copy. These examples use the key name prefix defined in ./object_name.json.",
value: choices.CONDITIONAL_COPY,
},
{
name: "Perform a conditional write. This example use the sample file ./text02.txt.",
value: choices.CONDITIONAL_WRITE,
},
{ name: "Finish the workflow.", value: choices.EXIT },
],
},
);
/**
* @param {S3Client} client
* @param {string[]} buckets
*/
const getAllFiles = async (client, buckets) => {
/** @type {{bucket: string, key: string, version: string}[]} */
const files = [];
for (const bucket of buckets) {
const objectsResponse = await client.send(
new ListObjectVersionsCommand({ Bucket: bucket }),
);
for (const version of objectsResponse.Versions || []) {
const { Key } = version;
files.push({ bucket, key: Key });
}
}
return files;
};
/**
* @param {S3Client} client
* @param {string[]} buckets
* @param {string} key
*/
const getEtag = async (client, bucket, key) => {
const objectsResponse = await client.send(
new GetObjectCommand({
Bucket: bucket,
Key: key,
}),
);
return objectsResponse.ETag;
};
/**
* @param {S3Client} client
* @param {string[]} buckets
*/
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
export const replAction = (scenarios, client) =>
new ScenarioAction(
"replAction",
async (state) => {
const files = await getAllFiles(client, [
state.sourceBucketName,
state.destinationBucketName,
]);
const fileInput = new scenarios.ScenarioInput(
"selectedFile",
"Select a file to use:",
{
type: "select",
choices: files.map((file, index) => ({
name: `${index + 1}: ${file.bucket}: ${file.key} (Etag: ${
file.version
})`,
value: index,
})),
},
);
const condReadOptions = new scenarios.ScenarioInput(
"selectOption",
"Which conditional read action would you like to take?",
{
type: "select",
choices: [
"If-Match: using the object's ETag. This condition should succeed.",
"If-None-Match: using the object's ETag. This condition should fail.",
"If-Modified-Since: using yesterday's date. This condition should succeed.",
"If-Unmodified-Since: using yesterday's date. This condition should fail.",
],
},
);
const condCopyOptions = new scenarios.ScenarioInput(
"selectOption",
"Which conditional copy action would you like to take?",
{
type: "select",
choices: [
"If-Match: using the object's ETag. This condition should succeed.",
"If-None-Match: using the object's ETag. This condition should fail.",
"If-Modified-Since: using yesterday's date. This condition should succeed.",
"If-Unmodified-Since: using yesterday's date. This condition should fail.",
],
},
);
const condWriteOptions = new scenarios.ScenarioInput(
"selectOption",
"Which conditional write action would you like to take?",
{
type: "select",
choices: [
"IfNoneMatch condition on the object key: If the key is a duplicate, the write will fail.",
],
},
);
const { replChoice } = state;
switch (replChoice) {
case choices.LIST_ALL_FILES: {
const files = await getAllFiles(client, [
state.sourceBucketName,
state.destinationBucketName,
]);
state.replOutput = files
.map(
(file) => `Items in bucket ${file.bucket}: object: ${file.key} `,
)
.join("\n");
break;
}
case choices.CONDITIONAL_READ:
{
const selectedCondRead = await condReadOptions.handle(state);
if (
selectedCondRead ===
"If-Match: using the object's ETag. This condition should succeed."
) {
const bucket = state.sourceBucketName;
const key = "file01.txt";
const ETag = await getEtag(client, bucket, key);
try {
await client.send(
new GetObjectCommand({
Bucket: bucket,
Key: key,
IfMatch: ETag,
}),
);
state.replOutput = `${key} in bucket ${state.sourceBucketName} read because ETag provided matches the object's ETag.`;
} catch (err) {
state.replOutput = `Unable to read object ${key} in bucket ${state.sourceBucketName}: ${err.message}`;
}
break;
}
if (
selectedCondRead ===
"If-None-Match: using the object's ETag. This condition should fail."
) {
const bucket = state.sourceBucketName;
const key = "file01.txt";
const ETag = await getEtag(client, bucket, key);
try {
await client.send(
new GetObjectCommand({
Bucket: bucket,
Key: key,
IfNoneMatch: ETag,
}),
);
state.replOutput = `${key} in ${state.sourceBucketName} was returned.`;
} catch (err) {
state.replOutput = `${key} in ${state.sourceBucketName} was not read: ${err.message}`;
}
break;
}
if (
selectedCondRead ===
"If-Modified-Since: using yesterday's date. This condition should succeed."
) {
const date = new Date();
date.setDate(date.getDate() - 1);
const bucket = state.sourceBucketName;
const key = "file01.txt";
try {
await client.send(
new GetObjectCommand({
Bucket: bucket,
Key: key,
IfModifiedSince: date,
}),
);
state.replOutput = `${key} in bucket ${state.sourceBucketName} read because it has been created or modified in the last 24 hours.`;
} catch (err) {
state.replOutput = `Unable to read object ${key} in bucket ${state.sourceBucketName}: ${err.message}`;
}
break;
}
if (
selectedCondRead ===
"If-Unmodified-Since: using yesterday's date. This condition should fail."
) {
const bucket = state.sourceBucketName;
const key = "file01.txt";
const date = new Date();
date.setDate(date.getDate() - 1);
try {
await client.send(
new GetObjectCommand({
Bucket: bucket,
Key: key,
IfUnmodifiedSince: date,
}),
);
state.replOutput = `${key} in ${state.sourceBucketName} was read.`;
} catch (err) {
state.replOutput = `${key} in ${state.sourceBucketName} was not read: ${err.message}`;
}
break;
}
}
break;
case choices.CONDITIONAL_COPY: {
const selectedCondCopy = await condCopyOptions.handle(state);
if (
selectedCondCopy ===
"If-Match: using the object's ETag. This condition should succeed."
) {
const bucket = state.sourceBucketName;
const key = "file01.txt";
const ETag = await getEtag(client, bucket, key);
const copySource = `${bucket}/${key}`;
// Optionally edit the default key name prefix of the copied object in ./object_name.json.
const name = data.name;
const copiedKey = `${name}${key}`;
try {
await client.send(
new CopyObjectCommand({
CopySource: copySource,
Bucket: state.destinationBucketName,
Key: copiedKey,
CopySourceIfMatch: ETag,
}),
);
state.replOutput = `${key} copied as ${copiedKey} to bucket ${state.destinationBucketName} because ETag provided matches the object's ETag.`;
} catch (err) {
state.replOutput = `Unable to copy object ${key} as ${copiedKey} to bucket ${state.destinationBucketName}: ${err.message}`;
}
break;
}
if (
selectedCondCopy ===
"If-None-Match: using the object's ETag. This condition should fail."
) {
const bucket = state.sourceBucketName;
const key = "file01.txt";
const ETag = await getEtag(client, bucket, key);
const copySource = `${bucket}/${key}`;
// Optionally edit the default key name prefix of the copied object in ./object_name.json.
const name = data.name;
const copiedKey = `${name}${key}`;
try {
await client.send(
new CopyObjectCommand({
CopySource: copySource,
Bucket: state.destinationBucketName,
Key: copiedKey,
CopySourceIfNoneMatch: ETag,
}),
);
state.replOutput = `${copiedKey} copied to bucket ${state.destinationBucketName}`;
} catch (err) {
state.replOutput = `Unable to copy object as ${key} as as ${copiedKey} to bucket ${state.destinationBucketName}: ${err.message}`;
}
break;
}
if (
selectedCondCopy ===
"If-Modified-Since: using yesterday's date. This condition should succeed."
) {
const bucket = state.sourceBucketName;
const key = "file01.txt";
const copySource = `${bucket}/${key}`;
// Optionally edit the default key name prefix of the copied object in ./object_name.json.
const name = data.name;
const copiedKey = `${name}${key}`;
const date = new Date();
date.setDate(date.getDate() - 1);
try {
await client.send(
new CopyObjectCommand({
CopySource: copySource,
Bucket: state.destinationBucketName,
Key: copiedKey,
CopySourceIfModifiedSince: date,
}),
);
state.replOutput = `${key} copied as ${copiedKey} to bucket ${state.destinationBucketName} because it has been created or modified in the last 24 hours.`;
} catch (err) {
state.replOutput = `Unable to copy object ${key} as ${copiedKey} to bucket ${state.destinationBucketName} : ${err.message}`;
}
break;
}
if (
selectedCondCopy ===
"If-Unmodified-Since: using yesterday's date. This condition should fail."
) {
const bucket = state.sourceBucketName;
const key = "file01.txt";
const copySource = `${bucket}/${key}`;
// Optionally edit the default key name prefix of the copied object in ./object_name.json.
const name = data.name;
const copiedKey = `${name}${key}`;
const date = new Date();
date.setDate(date.getDate() - 1);
try {
await client.send(
new CopyObjectCommand({
CopySource: copySource,
Bucket: state.destinationBucketName,
Key: copiedKey,
CopySourceIfUnmodifiedSince: date,
}),
);
state.replOutput = `${copiedKey} copied to bucket ${state.destinationBucketName} because it has not been created or modified in the last 24 hours.`;
} catch (err) {
state.replOutput = `Unable to copy object ${key} to bucket ${state.destinationBucketName}: ${err.message}`;
}
}
break;
}
case choices.CONDITIONAL_WRITE:
{
const selectedCondWrite = await condWriteOptions.handle(state);
if (
selectedCondWrite ===
"IfNoneMatch condition on the object key: If the key is a duplicate, the write will fail."
) {
// Optionally edit the default key name prefix of the copied object in ./object_name.json.
const key = "text02.txt";
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const filePath = path.join(__dirname, "text02.txt");
try {
await client.send(
new PutObjectCommand({
Bucket: `${state.destinationBucketName}`,
Key: `${key}`,
Body: await readFile(filePath),
IfNoneMatch: "*",
}),
);
state.replOutput = `${key} uploaded to bucket ${state.destinationBucketName} because the key is not a duplicate.`;
} catch (err) {
state.replOutput = `Unable to upload object to bucket ${state.destinationBucketName}:${err.message}`;
}
break;
}
}
break;
default:
throw new Error(`Invalid replChoice: ${replChoice}`);
}
},
{
whileConfig: {
whileFn: ({ replChoice }) => replChoice !== choices.EXIT,
input: replInput(scenarios),
output: new ScenarioOutput("REPL output", (state) => state.replOutput, {
preformatted: true,
}),
},
},
);
export { replInput, choices };
```
Destroy all created resources (clean.steps.js).
```
import {
DeleteObjectCommand,
DeleteBucketCommand,
ListObjectVersionsCommand,
} from "@aws-sdk/client-s3";
/**
* @typedef {import("@aws-doc-sdk-examples/lib/scenario/index.js")} Scenarios
*/
/**
* @typedef {import("@aws-sdk/client-s3").S3Client} S3Client
*/
/**
* @param {Scenarios} scenarios
*/
const confirmCleanup = (scenarios) =>
new scenarios.ScenarioInput("confirmCleanup", "Clean up resources?", {
type: "confirm",
});
/**
* @param {Scenarios} scenarios
* @param {S3Client} client
*/
const cleanupAction = (scenarios, client) =>
new scenarios.ScenarioAction("cleanupAction", async (state) => {
const { sourceBucketName, destinationBucketName } = state;
const buckets = [sourceBucketName, destinationBucketName].filter((b) => b);
for (const bucket of buckets) {
try {
let objectsResponse;
objectsResponse = await client.send(
new ListObjectVersionsCommand({
Bucket: bucket,
}),
);
for (const version of objectsResponse.Versions || []) {
const { Key, VersionId } = version;
try {
await client.send(
new DeleteObjectCommand({
Bucket: bucket,
Key,
VersionId,
}),
);
} catch (err) {
console.log(`An error occurred: ${err.message} `);
}
}
} catch (e) {
if (e instanceof Error && e.name === "NoSuchBucket") {
console.log("Objects and buckets have already been deleted.");
continue;
}
throw e;
}
await client.send(new DeleteBucketCommand({ Bucket: bucket }));
console.log(`Delete for ${bucket} complete.`);
}
});
export { confirmCleanup, cleanupAction };
```
+ For API details, see the following topics in *AWS SDK for JavaScript API Reference*.
+ [CopyObject](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/CopyObjectCommand)
+ [GetObject](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/GetObjectCommand)
+ [PutObject](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/PutObjectCommand)
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/s3/scenarios/conditional_requests#code-examples).
Run an interactive scenario demonstrating Amazon S3 conditional requests.
```
"""
Purpose
Shows how to use AWS SDK for Python (Boto3) to get started using conditional requests for
Amazon Simple Storage Service (Amazon S3).
"""
import logging
import random
import sys
import datetime
import boto3
from botocore.exceptions import ClientError
from s3_conditional_requests import S3ConditionalRequests
# Add relative path to include demo_tools in this code example without need for setup.
sys.path.append("../../../..")
import demo_tools.question as q # noqa
# Constants
FILE_CONTENT = "This is a test file for S3 conditional requests."
RANDOM_SUFFIX = str(random.randint(100, 999))
logger = logging.getLogger(__name__)
class ConditionalRequestsScenario:
"""Runs a scenario that shows how to use S3 Conditional Requests."""
def __init__(self, conditional_requests, s3_client):
"""
:param conditional_requests: An object that wraps S3 conditional request actions.
:param s3_client: A Boto3 S3 client for setup and cleanup operations.
"""
self.conditional_requests = conditional_requests
self.s3_client = s3_client
def setup_scenario(self, source_bucket: str, dest_bucket: str, object_key: str):
"""
Sets up the scenario by creating a source and destination bucket.
Prompts the user to provide a bucket name prefix.
:param source_bucket: The name of the source bucket.
:param dest_bucket: The name of the destination bucket.
:param object_key: The name of a test file to add to the source bucket.
"""
# Create the buckets.
try:
self.s3_client.create_bucket(Bucket=source_bucket)
self.s3_client.create_bucket(Bucket=dest_bucket)
print(
f"Created source bucket: {source_bucket} and destination bucket: {dest_bucket}"
)
except ClientError as e:
error_code = e.response["Error"]["Code"]
logger.error(f"Error creating buckets: {error_code}")
raise
# Upload test file into the source bucket.
try:
print(f"Uploading file {object_key} to bucket {source_bucket}")
response = self.s3_client.put_object(
Bucket=source_bucket, Key=object_key, Body=FILE_CONTENT
)
object_etag = response["ETag"]
return object_etag
except Exception as e:
logger.error(
f"Failed to upload file {object_key} to bucket {source_bucket}: {e}"
)
def cleanup_scenario(self, source_bucket: str, dest_bucket: str):
"""
Cleans up the scenario by deleting the source and destination buckets.
:param source_bucket: The name of the source bucket.
:param dest_bucket: The name of the destination bucket.
"""
self.cleanup_bucket(source_bucket)
self.cleanup_bucket(dest_bucket)
def cleanup_bucket(self, bucket_name: str):
"""
Cleans up the bucket by deleting all objects and then the bucket itself.
:param bucket_name: The name of the bucket.
"""
try:
# Get list of all objects in the bucket.
list_response = self.s3_client.list_objects_v2(Bucket=bucket_name)
objs = list_response.get("Contents", [])
for obj in objs:
key = obj["Key"]
self.s3_client.delete_object(Bucket=bucket_name, Key=key)
self.s3_client.delete_bucket(Bucket=bucket_name)
print(f"Cleaned up bucket: {bucket_name}.")
except ClientError as e:
error_code = e.response["Error"]["Code"]
if error_code == "NoSuchBucket":
logger.info(f"Bucket {bucket_name} does not exist, skipping cleanup.")
else:
logger.error(f"Error deleting bucket: {error_code}")
raise
def display_buckets(self, source_bucket: str, dest_bucket: str):
"""
Display a list of the objects in the test buckets.
:param source_bucket: The name of the source bucket.
:param dest_bucket: The name of the destination bucket.
"""
self.list_bucket_contents(source_bucket)
self.list_bucket_contents(dest_bucket)
def list_bucket_contents(self, bucket_name):
"""
Display a list of the objects in the bucket.
:param bucket_name: The name of the bucket.
"""
try:
# Get list of all objects in the bucket.
print(f"\t Items in bucket {bucket_name}")
list_response = self.s3_client.list_objects_v2(Bucket=bucket_name)
objs = list_response.get("Contents", [])
if not objs:
print("\t\tNo objects found.")
for obj in objs:
key = obj["Key"]
print(f"\t\t object: {key} ETag {obj['ETag']}")
return objs
except ClientError as e:
error_code = e.response["Error"]["Code"]
if error_code == "NoSuchBucket":
logger.info(f"Bucket {bucket_name} does not exist.")
else:
logger.error(f"Error listing bucket and objects: {error_code}")
raise
def display_menu(
self, source_bucket: str, dest_bucket: str, object_key: str, etag: str
):
"""
Displays the menu of conditional request options for the user.
:param source_bucket: The name of the source bucket.
:param dest_bucket: The name of the destination bucket.
:param object_key: The key of the test object in the source bucket.
:param etag: The etag of the test object in the source bucket.
"""
actions = [
"Print list of bucket items.",
"Perform a conditional read.",
"Perform a conditional copy.",
"Perform a conditional write.",
"Clean up and exit.",
]
conditions = [
"If-Match: using the object's ETag. This condition should succeed.",
"If-None-Match: using the object's ETag. This condition should fail.",
"If-Modified-Since: using yesterday's date. This condition should succeed.",
"If-Unmodified-Since: using yesterday's date. This condition should fail.",
]
condition_types = [
"IfMatch",
"IfNoneMatch",
"IfModifiedSince",
"IfUnmodifiedSince",
]
copy_condition_types = [
"CopySourceIfMatch",
"CopySourceIfNoneMatch",
"CopySourceIfModifiedSince",
"CopySourceIfUnmodifiedSince",
]
yesterday_date = datetime.datetime.utcnow() - datetime.timedelta(days=1)
choice = 0
while choice != 4:
print("-" * 88)
print("Choose an action to explore some example conditional requests.")
choice = q.choose("Which action would you like to take? ", actions)
if choice == 0:
print("Listing the objects and buckets.")
self.display_buckets(source_bucket, dest_bucket)
elif choice == 1:
print("Perform a conditional read.")
condition_type = q.choose("Enter the condition type : ", conditions)
if condition_type == 0 or condition_type == 1:
self.conditional_requests.get_object_conditional(
object_key, source_bucket, condition_types[condition_type], etag
)
elif condition_type == 2 or condition_type == 3:
self.conditional_requests.get_object_conditional(
object_key,
source_bucket,
condition_types[condition_type],
yesterday_date,
)
elif choice == 2:
print("Perform a conditional copy.")
condition_type = q.choose("Enter the condition type : ", conditions)
dest_key = q.ask("Enter an object key: ", q.non_empty)
if condition_type == 0 or condition_type == 1:
self.conditional_requests.copy_object_conditional(
object_key,
dest_key,
source_bucket,
dest_bucket,
copy_condition_types[condition_type],
etag,
)
elif condition_type == 2 or condition_type == 3:
self.conditional_requests.copy_object_conditional(
object_key,
dest_key,
copy_condition_types[condition_type],
yesterday_date,
)
elif choice == 3:
print(
"Perform a conditional write using IfNoneMatch condition on the object key."
)
print("If the key is a duplicate, the write will fail.")
object_key = q.ask("Enter an object key: ", q.non_empty)
self.conditional_requests.put_object_conditional(
object_key, source_bucket, b"Conditional write example data."
)
elif choice == 4:
print("Proceeding to cleanup.")
def run_scenario(self):
"""
Runs the interactive scenario.
"""
print("-" * 88)
print("Welcome to the Amazon S3 conditional requests example.")
print("-" * 88)
print(
f"""\
This example demonstrates the use of conditional requests for S3 operations.
You can use conditional requests to add preconditions to S3 read requests to return or copy
an object based on its Entity tag (ETag), or last modified date.
You can use a conditional write requests to prevent overwrites by ensuring
there is no existing object with the same key.
This example will allow you to perform conditional reads
and writes that will succeed or fail based on your selected options.
Sample buckets and a sample object will be created as part of the example.
"""
)
bucket_prefix = q.ask("Enter a bucket name prefix: ", q.non_empty)
source_bucket_name = f"{bucket_prefix}-source-{RANDOM_SUFFIX}"
dest_bucket_name = f"{bucket_prefix}-dest-{RANDOM_SUFFIX}"
object_key = "test-upload-file.txt"
try:
etag = self.setup_scenario(source_bucket_name, dest_bucket_name, object_key)
self.display_menu(source_bucket_name, dest_bucket_name, object_key, etag)
finally:
self.cleanup_scenario(source_bucket_name, dest_bucket_name)
print("-" * 88)
print("Thanks for watching.")
print("-" * 88)
if __name__ == "__main__":
scenario = ConditionalRequestsScenario(
S3ConditionalRequests.from_client(), boto3.client("s3")
)
scenario.run_scenario()
```
A wrapper class that defines the conditional request operations.
```
import boto3
import logging
from botocore.exceptions import ClientError
# Configure logging
logger = logging.getLogger(__name__)
class S3ConditionalRequests:
"""Encapsulates S3 conditional request operations."""
def __init__(self, s3_client):
self.s3 = s3_client
@classmethod
def from_client(cls):
"""
Instantiates this class from a Boto3 client.
"""
s3_client = boto3.client("s3")
return cls(s3_client)
def get_object_conditional(
self,
object_key: str,
source_bucket: str,
condition_type: str,
condition_value: str,
):
"""
Retrieves an object from Amazon S3 with a conditional request.
:param object_key: The key of the object to retrieve.
:param source_bucket: The source bucket of the object.
:param condition_type: The type of condition: 'IfMatch', 'IfNoneMatch', 'IfModifiedSince', 'IfUnmodifiedSince'.
:param condition_value: The value to use for the condition.
"""
try:
response = self.s3.get_object(
Bucket=source_bucket,
Key=object_key,
**{condition_type: condition_value},
)
sample_bytes = response["Body"].read(20)
print(
f"\tConditional read successful. Here are the first 20 bytes of the object:\n"
)
print(f"\t{sample_bytes}")
except ClientError as e:
error_code = e.response["Error"]["Code"]
if error_code == "PreconditionFailed":
print("\tConditional read failed: Precondition failed")
elif error_code == "304": # Not modified error code.
print("\tConditional read failed: Object not modified")
else:
logger.error(f"Unexpected error: {error_code}")
raise
def put_object_conditional(self, object_key: str, source_bucket: str, data: bytes):
"""
Uploads an object to Amazon S3 with a conditional request. Prevents overwrite
using an IfNoneMatch condition for the object key.
:param object_key: The key of the object to upload.
:param source_bucket: The source bucket of the object.
:param data: The data to upload.
"""
try:
self.s3.put_object(
Bucket=source_bucket, Key=object_key, Body=data, IfNoneMatch="*"
)
print(
f"\tConditional write successful for key {object_key} in bucket {source_bucket}."
)
except ClientError as e:
error_code = e.response["Error"]["Code"]
if error_code == "PreconditionFailed":
print("\tConditional write failed: Precondition failed")
else:
logger.error(f"Unexpected error: {error_code}")
raise
def copy_object_conditional(
self,
source_key: str,
dest_key: str,
source_bucket: str,
dest_bucket: str,
condition_type: str,
condition_value: str,
):
"""
Copies an object from one Amazon S3 bucket to another with a conditional request.
:param source_key: The key of the source object to copy.
:param dest_key: The key of the destination object.
:param source_bucket: The source bucket of the object.
:param dest_bucket: The destination bucket of the object.
:param condition_type: The type of condition to apply, e.g.
'CopySourceIfMatch', 'CopySourceIfNoneMatch', 'CopySourceIfModifiedSince', 'CopySourceIfUnmodifiedSince'.
:param condition_value: The value to use for the condition.
"""
try:
self.s3.copy_object(
Bucket=dest_bucket,
Key=dest_key,
CopySource={"Bucket": source_bucket, "Key": source_key},
**{condition_type: condition_value},
)
print(
f"\tConditional copy successful for key {dest_key} in bucket {dest_bucket}."
)
except ClientError as e:
error_code = e.response["Error"]["Code"]
if error_code == "PreconditionFailed":
print("\tConditional copy failed: Precondition failed")
elif error_code == "304": # Not modified error code.
print("\tConditional copy failed: Object not modified")
else:
logger.error(f"Unexpected error: {error_code}")
raise
```
+ For API details, see the following topics in *AWS SDK for Python (Boto3) API Reference*.
+ [CopyObject](https://docs.aws.amazon.com/goto/boto3/s3-2006-03-01/CopyObject)
+ [GetObject](https://docs.aws.amazon.com/goto/boto3/s3-2006-03-01/GetObject)
+ [PutObject](https://docs.aws.amazon.com/goto/boto3/s3-2006-03-01/PutObject)
------
# Manage access control lists (ACLs) for Amazon S3 buckets using an AWS SDK
The following code example shows how to manage access control lists (ACLs) for Amazon S3 buckets.
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/ManageACLsExample#code-examples).
```
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Amazon.S3;
using Amazon.S3.Model;
///
/// This example shows how to manage Amazon Simple Storage Service
/// (Amazon S3) access control lists (ACLs) to control Amazon S3 bucket
/// access.
///
public class ManageACLs
{
public static async Task Main()
{
string bucketName = "amzn-s3-demo-bucket1";
string newBucketName = "amzn-s3-demo-bucket2";
string keyName = "sample-object.txt";
string emailAddress = "someone@example.com";
// If the AWS Region where your bucket is located is different from
// the Region defined for the default user, pass the Amazon S3 bucket's
// name to the client constructor. It should look like this:
// RegionEndpoint bucketRegion = RegionEndpoint.USEast1;
IAmazonS3 client = new AmazonS3Client();
await TestBucketObjectACLsAsync(client, bucketName, newBucketName, keyName, emailAddress);
}
///
/// Creates a new Amazon S3 bucket with a canned ACL, then retrieves the ACL
/// information and then adds a new ACL to one of the objects in the
/// Amazon S3 bucket.
///
/// The initialized Amazon S3 client object used to call
/// methods to create a bucket, get an ACL, and add a different ACL to
/// one of the objects.
/// A string representing the original Amazon S3
/// bucket name.
/// A string representing the name of the
/// new bucket that will be created.
/// A string representing the key name of an Amazon S3
/// object for which we will change the ACL.
/// A string representing the email address
/// belonging to the person to whom access to the Amazon S3 bucket will be
/// granted.
public static async Task TestBucketObjectACLsAsync(
IAmazonS3 client,
string bucketName,
string newBucketName,
string keyName,
string emailAddress)
{
try
{
// Create a new Amazon S3 bucket and specify canned ACL.
var success = await CreateBucketWithCannedACLAsync(client, newBucketName);
// Get the ACL on a bucket.
await GetBucketACLAsync(client, bucketName);
// Add (replace) the ACL on an object in a bucket.
await AddACLToExistingObjectAsync(client, bucketName, keyName, emailAddress);
}
catch (AmazonS3Exception amazonS3Exception)
{
Console.WriteLine($"Exception: {amazonS3Exception.Message}");
}
}
///
/// Creates a new Amazon S3 bucket with a canned ACL attached.
///
/// The initialized client object used to call
/// PutBucketAsync.
/// A string representing the name of the
/// new Amazon S3 bucket.
/// Returns a boolean value indicating success or failure.
public static async Task CreateBucketWithCannedACLAsync(IAmazonS3 client, string newBucketName)
{
var request = new PutBucketRequest()
{
BucketName = newBucketName,
BucketRegion = S3Region.EUWest1,
// Add a canned ACL.
CannedACL = S3CannedACL.LogDeliveryWrite,
};
var response = await client.PutBucketAsync(request);
return response.HttpStatusCode == System.Net.HttpStatusCode.OK;
}
///
/// Retrieves the ACL associated with the Amazon S3 bucket name in the
/// bucketName parameter.
///
/// The initialized client object used to call
/// PutBucketAsync.
/// The Amazon S3 bucket for which we want to get the
/// ACL list.
/// Returns an S3AccessControlList returned from the call to
/// GetACLAsync.
public static async Task GetBucketACLAsync(IAmazonS3 client, string bucketName)
{
GetACLResponse response = await client.GetACLAsync(new GetACLRequest
{
BucketName = bucketName,
});
return response.AccessControlList;
}
///
/// Adds a new ACL to an existing object in the Amazon S3 bucket.
///
/// The initialized client object used to call
/// PutBucketAsync.
/// A string representing the name of the Amazon S3
/// bucket containing the object to which we want to apply a new ACL.
/// A string representing the name of the object
/// to which we want to apply the new ACL.
/// The email address of the person to whom
/// we will be applying to whom access will be granted.
public static async Task AddACLToExistingObjectAsync(IAmazonS3 client, string bucketName, string keyName, string emailAddress)
{
// Retrieve the ACL for an object.
GetACLResponse aclResponse = await client.GetACLAsync(new GetACLRequest
{
BucketName = bucketName,
Key = keyName,
});
S3AccessControlList acl = aclResponse.AccessControlList;
// Retrieve the owner.
Owner owner = acl.Owner;
// Clear existing grants.
acl.Grants.Clear();
// Add a grant to reset the owner's full permission
// (the previous clear statement removed all permissions).
var fullControlGrant = new S3Grant
{
Grantee = new S3Grantee { CanonicalUser = acl.Owner.Id },
};
acl.AddGrant(fullControlGrant.Grantee, S3Permission.FULL_CONTROL);
// Specify email to identify grantee for granting permissions.
var grantUsingEmail = new S3Grant
{
Grantee = new S3Grantee { EmailAddress = emailAddress },
Permission = S3Permission.WRITE_ACP,
};
// Specify log delivery group as grantee.
var grantLogDeliveryGroup = new S3Grant
{
Grantee = new S3Grantee { URI = "http://acs.amazonaws.com/groups/s3/LogDelivery" },
Permission = S3Permission.WRITE,
};
// Create a new ACL.
var newAcl = new S3AccessControlList
{
Grants = new List { grantUsingEmail, grantLogDeliveryGroup },
Owner = owner,
};
// Set the new ACL. We're throwing away the response here.
_ = await client.PutACLAsync(new PutACLRequest
{
BucketName = bucketName,
Key = keyName,
AccessControlList = newAcl,
});
}
}
```
+ For API details, see the following topics in *AWS SDK for .NET API Reference*.
+ [GetBucketAcl](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetBucketAcl)
+ [GetObjectAcl](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObjectAcl)
+ [PutBucketAcl](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/PutBucketAcl)
+ [PutObjectAcl](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/PutObjectAcl)
------
# Manage large Amazon SQS messages using Amazon S3 with an AWS SDK
The following code example shows how to use the Amazon SQS Extended Client Library to work with large Amazon SQS messages.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/sqs#code-examples).
```
import com.amazon.sqs.javamessaging.AmazonSQSExtendedClient;
import com.amazon.sqs.javamessaging.ExtendedClientConfiguration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.BucketLifecycleConfiguration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteBucketRequest;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.ExpirationStatus;
import software.amazon.awssdk.services.s3.model.LifecycleExpiration;
import software.amazon.awssdk.services.s3.model.LifecycleRule;
import software.amazon.awssdk.services.s3.model.LifecycleRuleFilter;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectVersionsResponse;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
import software.amazon.awssdk.services.s3.model.PutBucketLifecycleConfigurationRequest;
import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.CreateQueueRequest;
import software.amazon.awssdk.services.sqs.model.CreateQueueResponse;
import software.amazon.awssdk.services.sqs.model.DeleteMessageRequest;
import software.amazon.awssdk.services.sqs.model.DeleteQueueRequest;
import software.amazon.awssdk.services.sqs.model.Message;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageRequest;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageResponse;
import software.amazon.awssdk.services.sqs.model.SendMessageRequest;
import java.util.Arrays;
import java.util.List;
import java.util.UUID;
/**
* Example of using Amazon SQS Extended Client Library for Java 2.x.
*/
public class SqsExtendedClientExample {
private static final Logger logger = LoggerFactory.getLogger(SqsExtendedClientExample.class);
private String s3BucketName;
private String queueUrl;
private final String queueName;
private final S3Client s3Client;
private final SqsClient sqsExtendedClient;
private final int messageSize;
/**
* Constructor with default clients and message size.
*/
public SqsExtendedClientExample() {
this(S3Client.create(), 300000);
}
/**
* Constructor with custom S3 client and message size.
*
* @param s3Client The S3 client to use
* @param messageSize The size of the test message to create
*/
public SqsExtendedClientExample(S3Client s3Client, int messageSize) {
this.s3Client = s3Client;
this.messageSize = messageSize;
// Generate a unique bucket name.
this.s3BucketName = UUID.randomUUID() + "-" +
DateTimeFormat.forPattern("yyMMdd-hhmmss").print(new DateTime());
// Generate a unique queue name.
this.queueName = "MyQueue-" + UUID.randomUUID();
// Configure the SQS extended client.
final ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration()
.withPayloadSupportEnabled(s3Client, s3BucketName);
this.sqsExtendedClient = new AmazonSQSExtendedClient(SqsClient.builder().build(), extendedClientConfig);
}
public static void main(String[] args) {
SqsExtendedClientExample example = new SqsExtendedClientExample();
try {
example.setup();
example.sendAndReceiveMessage();
} finally {
example.cleanup();
}
}
/**
* Send a large message and receive it back.
*
* @return The received message
*/
public Message sendAndReceiveMessage() {
try {
// Create a large message.
char[] chars = new char[messageSize];
Arrays.fill(chars, 'x');
String largeMessage = new String(chars);
// Send the message.
final SendMessageRequest sendMessageRequest = SendMessageRequest.builder()
.queueUrl(queueUrl)
.messageBody(largeMessage)
.build();
sqsExtendedClient.sendMessage(sendMessageRequest);
logger.info("Sent message of size: {}", largeMessage.length());
// Receive and return the message.
final ReceiveMessageResponse receiveMessageResponse = sqsExtendedClient.receiveMessage(
ReceiveMessageRequest.builder().queueUrl(queueUrl).build());
List messages = receiveMessageResponse.messages();
if (messages.isEmpty()) {
throw new RuntimeException("No messages received");
}
Message message = messages.getFirst();
logger.info("\nMessage received.");
logger.info(" ID: {}", message.messageId());
logger.info(" Receipt handle: {}", message.receiptHandle());
logger.info(" Message body size: {}", message.body().length());
logger.info(" Message body (first 5 characters): {}", message.body().substring(0, 5));
return message;
} catch (RuntimeException e) {
logger.error("Error during message processing: {}", e.getMessage(), e);
throw e;
}
}
```
+ For more information, see [AWS SDK for Java 2.x Developer Guide](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html).
+ For API details, see the following topics in *AWS SDK for Java 2.x API Reference*.
+ [CreateBucket](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/CreateBucket)
+ [PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutBucketLifecycleConfiguration)
+ [ReceiveMessage](https://docs.aws.amazon.com/goto/SdkForJavaV2/sqs-2012-11-05/ReceiveMessage)
+ [SendMessage](https://docs.aws.amazon.com/goto/SdkForJavaV2/sqs-2012-11-05/SendMessage)
------
# Manage versioned Amazon S3 objects in batches with a Lambda function using an AWS SDK
The following code example shows how to manage versioned S3 objects in batches with a Lambda function.
------
#### [ Python ]
**SDK for Python (Boto3)**
Shows how to manipulate Amazon Simple Storage Service (Amazon S3) versioned objects in batches by creating jobs that call AWS Lambda functions to perform processing. This example creates a version-enabled bucket, uploads the stanzas from the poem *You Are Old, Father William* by Lewis Carroll, and uses Amazon S3 batch jobs to twist the poem in various ways.
**Learn how to:**
+ Create Lambda functions that operate on versioned objects.
+ Create a manifest of objects to update.
+ Create batch jobs that invoke Lambda functions to update objects.
+ Delete Lambda functions.
+ Empty and delete a versioned bucket.
This example is best viewed on GitHub. For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/s3/s3_versioning#batch-operation-demo).
**Services used in this example**
+ Amazon S3
------
# Parse Amazon S3 URIs using an AWS SDK
The following code example shows how to parse Amazon S3 URIs to extract important components like the bucket name and object key.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
Parse an Amazon S3 URI by using the [S3Uri](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Uri.html) class.
```
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Uri;
import software.amazon.awssdk.services.s3.S3Utilities;
import java.net.URI;
import java.util.List;
import java.util.Map;
/**
*
* @param s3Client - An S3Client through which you acquire an S3Uri instance.
* @param s3ObjectUrl - A complex URL (String) that is used to demonstrate S3Uri
* capabilities.
*/
public static void parseS3UriExample(S3Client s3Client, String s3ObjectUrl) {
logger.info(s3ObjectUrl);
// Console output:
// 'https://s3.us-west-1.amazonaws.com/myBucket/resources/doc.txt?versionId=abc123&partNumber=77&partNumber=88'.
// Create an S3Utilities object using the configuration of the s3Client.
S3Utilities s3Utilities = s3Client.utilities();
// From a String URL create a URI object to pass to the parseUri() method.
URI uri = URI.create(s3ObjectUrl);
S3Uri s3Uri = s3Utilities.parseUri(uri);
// If the URI contains no value for the Region, bucket or key, the SDK returns
// an empty Optional.
// The SDK returns decoded URI values.
Region region = s3Uri.region().orElse(null);
log("region", region);
// Console output: 'region: us-west-1'.
String bucket = s3Uri.bucket().orElse(null);
log("bucket", bucket);
// Console output: 'bucket: myBucket'.
String key = s3Uri.key().orElse(null);
log("key", key);
// Console output: 'key: resources/doc.txt'.
Boolean isPathStyle = s3Uri.isPathStyle();
log("isPathStyle", isPathStyle);
// Console output: 'isPathStyle: true'.
// If the URI contains no query parameters, the SDK returns an empty map.
Map> queryParams = s3Uri.rawQueryParameters();
log("rawQueryParameters", queryParams);
// Console output: 'rawQueryParameters: {versionId=[abc123], partNumber=[77,
// 88]}'.
// Retrieve the first or all values for a query parameter as shown in the
// following code.
String versionId = s3Uri.firstMatchingRawQueryParameter("versionId").orElse(null);
log("firstMatchingRawQueryParameter-versionId", versionId);
// Console output: 'firstMatchingRawQueryParameter-versionId: abc123'.
String partNumber = s3Uri.firstMatchingRawQueryParameter("partNumber").orElse(null);
log("firstMatchingRawQueryParameter-partNumber", partNumber);
// Console output: 'firstMatchingRawQueryParameter-partNumber: 77'.
List partNumbers = s3Uri.firstMatchingRawQueryParameters("partNumber");
log("firstMatchingRawQueryParameter", partNumbers);
// Console output: 'firstMatchingRawQueryParameter: [77, 88]'.
/*
* Object keys and query parameters with reserved or unsafe characters, must be
* URL-encoded.
* For example replace whitespace " " with "%20".
* Valid:
* "https://s3.us-west-1.amazonaws.com/myBucket/object%20key?query=%5Bbrackets%5D"
* Invalid:
* "https://s3.us-west-1.amazonaws.com/myBucket/object key?query=[brackets]"
*
* Virtual-hosted-style URIs with bucket names that contain a dot, ".", the dot
* must not be URL-encoded.
* Valid: "https://my.Bucket.s3.us-west-1.amazonaws.com/key"
* Invalid: "https://my%2EBucket.s3.us-west-1.amazonaws.com/key"
*/
}
private static void log(String s3UriElement, Object element) {
if (element == null) {
logger.info("{}: {}", s3UriElement, "null");
} else {
logger.info("{}: {}", s3UriElement, element);
}
}
```
------
# Perform a multipart copy of an Amazon S3 object using an AWS SDK
The following code example shows how to perform a multipart copy of an Amazon S3 object.
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/MPUapiCopyObjExample#code-examples).
```
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Amazon.S3;
using Amazon.S3.Model;
///
/// This example shows how to perform a multi-part copy from one Amazon
/// Simple Storage Service (Amazon S3) bucket to another.
///
public class MPUapiCopyObj
{
private const string SourceBucket = "amzn-s3-demo-bucket1";
private const string TargetBucket = "amzn-s3-demo-bucket2";
private const string SourceObjectKey = "example.mov";
private const string TargetObjectKey = "copied_video_file.mov";
///
/// This method starts the multi-part upload.
///
public static async Task Main()
{
var s3Client = new AmazonS3Client();
Console.WriteLine("Copying object...");
await MPUCopyObjectAsync(s3Client);
}
///
/// This method uses the passed client object to perform a multipart
/// copy operation.
///
/// An Amazon S3 client object that will be used
/// to perform the copy.
public static async Task MPUCopyObjectAsync(AmazonS3Client client)
{
// Create a list to store the copy part responses.
var copyResponses = new List();
// Setup information required to initiate the multipart upload.
var initiateRequest = new InitiateMultipartUploadRequest
{
BucketName = TargetBucket,
Key = TargetObjectKey,
};
// Initiate the upload.
InitiateMultipartUploadResponse initResponse =
await client.InitiateMultipartUploadAsync(initiateRequest);
// Save the upload ID.
string uploadId = initResponse.UploadId;
try
{
// Get the size of the object.
var metadataRequest = new GetObjectMetadataRequest
{
BucketName = SourceBucket,
Key = SourceObjectKey,
};
GetObjectMetadataResponse metadataResponse =
await client.GetObjectMetadataAsync(metadataRequest);
var objectSize = metadataResponse.ContentLength; // Length in bytes.
// Copy the parts.
var partSize = 5 * (long)Math.Pow(2, 20); // Part size is 5 MB.
long bytePosition = 0;
for (int i = 1; bytePosition < objectSize; i++)
{
var copyRequest = new CopyPartRequest
{
DestinationBucket = TargetBucket,
DestinationKey = TargetObjectKey,
SourceBucket = SourceBucket,
SourceKey = SourceObjectKey,
UploadId = uploadId,
FirstByte = bytePosition,
LastByte = bytePosition + partSize - 1 >= objectSize ? objectSize - 1 : bytePosition + partSize - 1,
PartNumber = i,
};
copyResponses.Add(await client.CopyPartAsync(copyRequest));
bytePosition += partSize;
}
// Set up to complete the copy.
var completeRequest = new CompleteMultipartUploadRequest
{
BucketName = TargetBucket,
Key = TargetObjectKey,
UploadId = initResponse.UploadId,
};
completeRequest.AddPartETags(copyResponses);
// Complete the copy.
CompleteMultipartUploadResponse completeUploadResponse =
await client.CompleteMultipartUploadAsync(completeRequest);
}
catch (AmazonS3Exception e)
{
Console.WriteLine($"Error encountered on server. Message:'{e.Message}' when writing an object");
}
catch (Exception e)
{
Console.WriteLine($"Unknown encountered on server. Message:'{e.Message}' when writing an object");
}
}
}
```
+ For API details, see the following topics in *AWS SDK for .NET API Reference*.
+ [CompleteMultipartUpload](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/CompleteMultipartUpload)
+ [CreateMultipartUpload](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/CreateMultipartUpload)
+ [GetObjectMetadata](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/GetObjectMetadata)
+ [UploadPartCopy](https://docs.aws.amazon.com/goto/DotNetSDKV3/s3-2006-03-01/UploadPartCopy)
------
# Receive and process Amazon S3 event notifications by using an AWS SDK
The following code example shows how to work with S3 event notifications in an object-oriented way.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
This example show how to process S3 notification event by using Amazon SQS.
```
/**
* This method receives S3 event notifications by using an SqsAsyncClient.
* After the client receives the messages it deserializes the JSON payload and logs them. It uses
* the S3EventNotification class (part of the S3 event notification API for Java) to deserialize
* the JSON payload and access the messages in an object-oriented way.
*
* @param queueUrl The URL of the AWS SQS queue that receives the S3 event notifications.
* @see S3EventNotification API.
*
* To use S3 event notification serialization/deserialization to objects, add the following
* dependency to your Maven pom.xml file.
*
* software.amazon.awssdk
* s3-event-notifications
*
*
*
* The S3 event notification API became available with version 2.25.11 of the Java SDK.
*
* This example shows the use of the API with AWS SQS, but it can be used to process S3 event notifications
* in AWS SNS or AWS Lambda as well.
*
* Note: The S3EventNotification class does not work with messages routed through AWS EventBridge.
*/
static void processS3Events(String bucketName, String queueUrl, String queueArn) {
try {
// Configure the bucket to send Object Created and Object Tagging notifications to an existing SQS queue.
s3Client.putBucketNotificationConfiguration(b -> b
.notificationConfiguration(ncb -> ncb
.queueConfigurations(qcb -> qcb
.events(Event.S3_OBJECT_CREATED, Event.S3_OBJECT_TAGGING)
.queueArn(queueArn)))
.bucket(bucketName)
).join();
triggerS3EventNotifications(bucketName);
// Wait for event notifications to propagate.
Thread.sleep(Duration.ofSeconds(5).toMillis());
boolean didReceiveMessages = true;
while (didReceiveMessages) {
// Display the number of messages that are available in the queue.
sqsClient.getQueueAttributes(b -> b
.queueUrl(queueUrl)
.attributeNames(QueueAttributeName.APPROXIMATE_NUMBER_OF_MESSAGES)
).thenAccept(attributeResponse ->
logger.info("Approximate number of messages in the queue: {}",
attributeResponse.attributes().get(QueueAttributeName.APPROXIMATE_NUMBER_OF_MESSAGES)))
.join();
// Receive the messages.
ReceiveMessageResponse response = sqsClient.receiveMessage(b -> b
.queueUrl(queueUrl)
).get();
logger.info("Count of received messages: {}", response.messages().size());
didReceiveMessages = !response.messages().isEmpty();
// Create a collection to hold the received message for deletion
// after we log the messages.
HashSet messagesToDelete = new HashSet<>();
// Process each message.
response.messages().forEach(message -> {
logger.info("Message id: {}", message.messageId());
// Deserialize JSON message body to a S3EventNotification object
// to access messages in an object-oriented way.
S3EventNotification event = S3EventNotification.fromJson(message.body());
// Log the S3 event notification record details.
if (event.getRecords() != null) {
event.getRecords().forEach(record -> {
String eventName = record.getEventName();
String key = record.getS3().getObject().getKey();
logger.info(record.toString());
logger.info("Event name is {} and key is {}", eventName, key);
});
}
// Add logged messages to collection for batch deletion.
messagesToDelete.add(DeleteMessageBatchRequestEntry.builder()
.id(message.messageId())
.receiptHandle(message.receiptHandle())
.build());
});
// Delete messages.
if (!messagesToDelete.isEmpty()) {
sqsClient.deleteMessageBatch(DeleteMessageBatchRequest.builder()
.queueUrl(queueUrl)
.entries(messagesToDelete)
.build()
).join();
}
} // End of while block.
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
}
```
+ For API details, see the following topics in *AWS SDK for Java 2.x API Reference*.
+ [DeleteMessageBatch](https://docs.aws.amazon.com/goto/SdkForJavaV2/sqs-2012-11-05/DeleteMessageBatch)
+ [GetQueueAttributes](https://docs.aws.amazon.com/goto/SdkForJavaV2/sqs-2012-11-05/GetQueueAttributes)
+ [PutBucketNotificationConfiguration](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutBucketNotificationConfiguration)
+ [ReceiveMessage](https://docs.aws.amazon.com/goto/SdkForJavaV2/sqs-2012-11-05/ReceiveMessage)
------
# Save EXIF and other image information using an AWS SDK
The following code example shows how to:
+ Get EXIF information from a a JPG, JPEG, or PNG file.
+ Upload the image file to an Amazon S3 bucket.
+ Use Amazon Rekognition to identify the three top attributes (labels) in the file.
+ Add the EXIF and label information to an Amazon DynamoDB table in the Region.
------
#### [ Rust ]
**SDK for Rust**
Get EXIF information from a JPG, JPEG, or PNG file, upload the image file to an Amazon S3 bucket, use Amazon Rekognition to identify the three top attributes (*labels* in Amazon Rekognition) in the file, and add the EXIF and label information to a Amazon DynamoDB table in the Region.
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/rustv1/cross_service/detect_labels/src/main.rs).
**Services used in this example**
+ DynamoDB
+ Amazon Rekognition
+ Amazon S3
------
# Send S3 event notifications to Amazon EventBridge using an AWS SDK
The following code example shows how to enable a bucket to send S3 event notifications to EventBridge and route notifications to an Amazon SNS topic and Amazon SQS queue.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
```
/** This method configures a bucket to send events to AWS EventBridge and creates a rule
* to route the S3 object created events to a topic and a queue.
*
* @param bucketName Name of existing bucket
* @param topicArn ARN of existing topic to receive S3 event notifications
* @param queueArn ARN of existing queue to receive S3 event notifications
*
* An AWS CloudFormation stack sets up the bucket, queue, topic before the method runs.
*/
public static String setBucketNotificationToEventBridge(String bucketName, String topicArn, String queueArn) {
try {
// Enable bucket to emit S3 Event notifications to EventBridge.
s3Client.putBucketNotificationConfiguration(b -> b
.bucket(bucketName)
.notificationConfiguration(b1 -> b1
.eventBridgeConfiguration(
SdkBuilder::build)
).build()).join();
// Create an EventBridge rule to route Object Created notifications.
PutRuleRequest putRuleRequest = PutRuleRequest.builder()
.name(RULE_NAME)
.eventPattern("""
{
"source": ["aws.s3"],
"detail-type": ["Object Created"],
"detail": {
"bucket": {
"name": ["%s"]
}
}
}
""".formatted(bucketName))
.build();
// Add the rule to the default event bus.
PutRuleResponse putRuleResponse = eventBridgeClient.putRule(putRuleRequest)
.whenComplete((r, t) -> {
if (t != null) {
logger.error("Error creating event bus rule: " + t.getMessage(), t);
throw new RuntimeException(t.getCause().getMessage(), t);
}
logger.info("Event bus rule creation request sent successfully. ARN is: {}", r.ruleArn());
}).join();
// Add the existing SNS topic and SQS queue as targets to the rule.
eventBridgeClient.putTargets(b -> b
.eventBusName("default")
.rule(RULE_NAME)
.targets(List.of (
Target.builder()
.arn(queueArn)
.id("Queue")
.build(),
Target.builder()
.arn(topicArn)
.id("Topic")
.build())
)
).join();
return putRuleResponse.ruleArn();
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
return null;
}
```
+ For API details, see the following topics in *AWS SDK for Java 2.x API Reference*.
+ [PutBucketNotificationConfiguration](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutBucketNotificationConfiguration)
+ [PutRule](https://docs.aws.amazon.com/goto/SdkForJavaV2/eventbridge-2015-10-07/PutRule)
+ [PutTargets](https://docs.aws.amazon.com/goto/SdkForJavaV2/eventbridge-2015-10-07/PutTargets)
------
# Track an Amazon S3 object upload or download using an AWS SDK
The following code example shows how to track an Amazon S3 object upload or download.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
Track the progress of a file upload.
```
public void trackUploadFile(S3TransferManager transferManager, String bucketName,
String key, URI filePathURI) {
UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
.putObjectRequest(b -> b.bucket(bucketName).key(key))
.addTransferListener(LoggingTransferListener.create()) // Add listener.
.source(Paths.get(filePathURI))
.build();
FileUpload fileUpload = transferManager.uploadFile(uploadFileRequest);
fileUpload.completionFuture().join();
/*
The SDK provides a LoggingTransferListener implementation of the TransferListener interface.
You can also implement the interface to provide your own logic.
Configure log4J2 with settings such as the following.
Log4J2 logs the progress. The following is example output for a 21.3 MB file upload.
Transfer initiated...
| | 0.0%
|==== | 21.1%
|============ | 60.5%
|====================| 100.0%
Transfer complete!
*/
}
```
Track the progress of a file download.
```
public void trackDownloadFile(S3TransferManager transferManager, String bucketName,
String key, String downloadedFileWithPath) {
DownloadFileRequest downloadFileRequest = DownloadFileRequest.builder()
.getObjectRequest(b -> b.bucket(bucketName).key(key))
.addTransferListener(LoggingTransferListener.create()) // Add listener.
.destination(Paths.get(downloadedFileWithPath))
.build();
FileDownload downloadFile = transferManager.downloadFile(downloadFileRequest);
CompletedFileDownload downloadResult = downloadFile.completionFuture().join();
/*
The SDK provides a LoggingTransferListener implementation of the TransferListener interface.
You can also implement the interface to provide your own logic.
Configure log4J2 with settings such as the following.
Log4J2 logs the progress. The following is example output for a 21.3 MB file download.
Transfer initiated...
|======= | 39.4%
|=============== | 78.8%
|====================| 100.0%
Transfer complete!
*/
}
```
+ For API details, see the following topics in *AWS SDK for Java 2.x API Reference*.
+ [GetObject](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/GetObject)
+ [PutObject](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/PutObject)
------
# Transform data for your application with S3 Object Lambda
The following code example shows how to transform data for your application with S3 Object Lambda.
------
#### [ .NET ]
**SDK for .NET**
Shows how to add custom code to standard S3 GET requests to modify the requested object retrieved from S3 so that the object suit the needs of the requesting client or application.
For complete source code and instructions on how to set up and run, see the full example on [GitHub](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/cross-service/S3ObjectLambdaFunction).
**Services used in this example**
+ Lambda
+ Amazon S3
------
# Example approaches for unit and integration testing with an AWS SDK
The following code example shows how to examples for best-practice techniques when writing unit and integration tests using an AWS SDK.
------
#### [ Rust ]
**SDK for Rust**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/rustv1/examples/testing#code-examples).
Cargo.toml for testing examples.
```
[package]
name = "testing-examples"
version = "0.1.0"
authors = [
"John Disanti ",
"Doug Schwartz ",
]
edition = "2021"
[dependencies]
async-trait = "0.1.51"
aws-config = { version = "1.0.1", features = ["behavior-version-latest"] }
aws-credential-types = { version = "1.0.1", features = [ "hardcoded-credentials", ] }
aws-sdk-s3 = { version = "1.4.0" }
aws-smithy-types = { version = "1.0.1" }
aws-smithy-runtime = { version = "1.0.1", features = ["test-util"] }
aws-smithy-runtime-api = { version = "1.0.1", features = ["test-util"] }
aws-types = { version = "1.0.1" }
clap = { version = "4.4", features = ["derive"] }
http = "0.2.9"
mockall = "0.11.4"
serde_json = "1"
tokio = { version = "1.20.1", features = ["full"] }
tracing-subscriber = { version = "0.3.15", features = ["env-filter"] }
[[bin]]
name = "main"
path = "src/main.rs"
```
Unit testing example using automock and a service wrapper.
```
use aws_sdk_s3 as s3;
#[allow(unused_imports)]
use mockall::automock;
use s3::operation::list_objects_v2::{ListObjectsV2Error, ListObjectsV2Output};
#[cfg(test)]
pub use MockS3Impl as S3;
#[cfg(not(test))]
pub use S3Impl as S3;
#[allow(dead_code)]
pub struct S3Impl {
inner: s3::Client,
}
#[cfg_attr(test, automock)]
impl S3Impl {
#[allow(dead_code)]
pub fn new(inner: s3::Client) -> Self {
Self { inner }
}
#[allow(dead_code)]
pub async fn list_objects(
&self,
bucket: &str,
prefix: &str,
continuation_token: Option,
) -> Result> {
self.inner
.list_objects_v2()
.bucket(bucket)
.prefix(prefix)
.set_continuation_token(continuation_token)
.send()
.await
}
}
#[allow(dead_code)]
pub async fn determine_prefix_file_size(
// Now we take a reference to our trait object instead of the S3 client
// s3_list: ListObjectsService,
s3_list: S3,
bucket: &str,
prefix: &str,
) -> Result {
let mut next_token: Option = None;
let mut total_size_bytes = 0;
loop {
let result = s3_list
.list_objects(bucket, prefix, next_token.take())
.await?;
// Add up the file sizes we got back
for object in result.contents() {
total_size_bytes += object.size().unwrap_or(0) as usize;
}
// Handle pagination, and break the loop if there are no more pages
next_token = result.next_continuation_token.clone();
if next_token.is_none() {
break;
}
}
Ok(total_size_bytes)
}
#[cfg(test)]
mod test {
use super::*;
use mockall::predicate::eq;
#[tokio::test]
async fn test_single_page() {
let mut mock = MockS3Impl::default();
mock.expect_list_objects()
.with(eq("test-bucket"), eq("test-prefix"), eq(None))
.return_once(|_, _, _| {
Ok(ListObjectsV2Output::builder()
.set_contents(Some(vec![
// Mock content for ListObjectsV2 response
s3::types::Object::builder().size(5).build(),
s3::types::Object::builder().size(2).build(),
]))
.build())
});
// Run the code we want to test with it
let size = determine_prefix_file_size(mock, "test-bucket", "test-prefix")
.await
.unwrap();
// Verify we got the correct total size back
assert_eq!(7, size);
}
#[tokio::test]
async fn test_multiple_pages() {
// Create the Mock instance with two pages of objects now
let mut mock = MockS3Impl::default();
mock.expect_list_objects()
.with(eq("test-bucket"), eq("test-prefix"), eq(None))
.return_once(|_, _, _| {
Ok(ListObjectsV2Output::builder()
.set_contents(Some(vec![
// Mock content for ListObjectsV2 response
s3::types::Object::builder().size(5).build(),
s3::types::Object::builder().size(2).build(),
]))
.set_next_continuation_token(Some("next".to_string()))
.build())
});
mock.expect_list_objects()
.with(
eq("test-bucket"),
eq("test-prefix"),
eq(Some("next".to_string())),
)
.return_once(|_, _, _| {
Ok(ListObjectsV2Output::builder()
.set_contents(Some(vec![
// Mock content for ListObjectsV2 response
s3::types::Object::builder().size(3).build(),
s3::types::Object::builder().size(9).build(),
]))
.build())
});
// Run the code we want to test with it
let size = determine_prefix_file_size(mock, "test-bucket", "test-prefix")
.await
.unwrap();
assert_eq!(19, size);
}
}
```
Integration testing example using StaticReplayClient.
```
use aws_sdk_s3 as s3;
#[allow(dead_code)]
pub async fn determine_prefix_file_size(
// Now we take a reference to our trait object instead of the S3 client
// s3_list: ListObjectsService,
s3: s3::Client,
bucket: &str,
prefix: &str,
) -> Result {
let mut next_token: Option = None;
let mut total_size_bytes = 0;
loop {
let result = s3
.list_objects_v2()
.prefix(prefix)
.bucket(bucket)
.set_continuation_token(next_token.take())
.send()
.await?;
// Add up the file sizes we got back
for object in result.contents() {
total_size_bytes += object.size().unwrap_or(0) as usize;
}
// Handle pagination, and break the loop if there are no more pages
next_token = result.next_continuation_token.clone();
if next_token.is_none() {
break;
}
}
Ok(total_size_bytes)
}
#[allow(dead_code)]
fn make_s3_test_credentials() -> s3::config::Credentials {
s3::config::Credentials::new(
"ATESTCLIENT",
"astestsecretkey",
Some("atestsessiontoken".to_string()),
None,
"",
)
}
#[cfg(test)]
mod test {
use super::*;
use aws_config::BehaviorVersion;
use aws_sdk_s3 as s3;
use aws_smithy_runtime::client::http::test_util::{ReplayEvent, StaticReplayClient};
use aws_smithy_types::body::SdkBody;
#[tokio::test]
async fn test_single_page() {
let page_1 = ReplayEvent::new(
http::Request::builder()
.method("GET")
.uri("https://test-bucket.s3.us-east-1.amazonaws.com/?list-type=2&prefix=test-prefix")
.body(SdkBody::empty())
.unwrap(),
http::Response::builder()
.status(200)
.body(SdkBody::from(include_str!("./testing/response_1.xml")))
.unwrap(),
);
let replay_client = StaticReplayClient::new(vec![page_1]);
let client: s3::Client = s3::Client::from_conf(
s3::Config::builder()
.behavior_version(BehaviorVersion::latest())
.credentials_provider(make_s3_test_credentials())
.region(s3::config::Region::new("us-east-1"))
.http_client(replay_client.clone())
.build(),
);
// Run the code we want to test with it
let size = determine_prefix_file_size(client, "test-bucket", "test-prefix")
.await
.unwrap();
// Verify we got the correct total size back
assert_eq!(7, size);
replay_client.assert_requests_match(&[]);
}
#[tokio::test]
async fn test_multiple_pages() {
let page_1 = ReplayEvent::new(
http::Request::builder()
.method("GET")
.uri("https://test-bucket.s3.us-east-1.amazonaws.com/?list-type=2&prefix=test-prefix")
.body(SdkBody::empty())
.unwrap(),
http::Response::builder()
.status(200)
.body(SdkBody::from(include_str!("./testing/response_multi_1.xml")))
.unwrap(),
);
let page_2 = ReplayEvent::new(
http::Request::builder()
.method("GET")
.uri("https://test-bucket.s3.us-east-1.amazonaws.com/?list-type=2&prefix=test-prefix&continuation-token=next")
.body(SdkBody::empty())
.unwrap(),
http::Response::builder()
.status(200)
.body(SdkBody::from(include_str!("./testing/response_multi_2.xml")))
.unwrap(),
);
let replay_client = StaticReplayClient::new(vec![page_1, page_2]);
let client: s3::Client = s3::Client::from_conf(
s3::Config::builder()
.behavior_version(BehaviorVersion::latest())
.credentials_provider(make_s3_test_credentials())
.region(s3::config::Region::new("us-east-1"))
.http_client(replay_client.clone())
.build(),
);
// Run the code we want to test with it
let size = determine_prefix_file_size(client, "test-bucket", "test-prefix")
.await
.unwrap();
assert_eq!(19, size);
replay_client.assert_requests_match(&[]);
}
}
```
------
# Recursively upload a local directory to an Amazon Simple Storage Service (Amazon S3) bucket
The following code example shows how to upload a local directory recursively to an Amazon Simple Storage Service (Amazon S3) bucket.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
Use an [S3TransferManager](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/S3TransferManager.html) to [upload a local directory](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/S3TransferManager.html#uploadDirectory(software.amazon.awssdk.transfer.s3.UploadDirectoryRequest)). View the [complete file](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/s3/src/main/java/com/example/s3/transfermanager/UploadADirectory.java) and [test](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/s3/src/test/java/TransferManagerTest.java).
```
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.services.s3.model.ObjectIdentifier;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.CompletedDirectoryUpload;
import software.amazon.awssdk.transfer.s3.model.DirectoryUpload;
import software.amazon.awssdk.transfer.s3.model.UploadDirectoryRequest;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.file.Paths;
import java.util.UUID;
public Integer uploadDirectory(S3TransferManager transferManager,
URI sourceDirectory, String bucketName) {
DirectoryUpload directoryUpload = transferManager.uploadDirectory(UploadDirectoryRequest.builder()
.source(Paths.get(sourceDirectory))
.bucket(bucketName)
.build());
CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join();
completedDirectoryUpload.failedTransfers()
.forEach(fail -> logger.warn("Object [{}] failed to transfer", fail.toString()));
return completedDirectoryUpload.failedTransfers().size();
}
```
+ For API details, see [UploadDirectory](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/UploadDirectory) in *AWS SDK for Java 2.x API Reference*.
------
# Upload or download large files to and from Amazon S3 using an AWS SDK
The following code examples show how to upload or download large files to and from Amazon S3.
For more information, see [Uploading an object using multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-upload-object.html).
------
#### [ .NET ]
**SDK for .NET**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/S3/#code-examples).
Call functions that transfer files to and from an S3 bucket using the Amazon S3 TransferUtility.
```
global using System.Text;
global using Amazon.S3;
global using Amazon.S3.Model;
global using Amazon.S3.Transfer;
global using TransferUtilityBasics;
// This Amazon S3 client uses the default user credentials
// defined for this computer.
using Microsoft.Extensions.Configuration;
IAmazonS3 client = new AmazonS3Client();
var transferUtil = new TransferUtility(client);
IConfiguration _configuration;
_configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("settings.json") // Load test settings from JSON file.
.AddJsonFile("settings.local.json",
true) // Optionally load local settings.
.Build();
// Edit the values in settings.json to use an S3 bucket and files that
// exist on your AWS account and on the local computer where you
// run this scenario.
var bucketName = _configuration["BucketName"];
var localPath = $"{Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData)}\\TransferFolder";
DisplayInstructions();
PressEnter();
Console.WriteLine();
// Upload a single file to an S3 bucket.
DisplayTitle("Upload a single file");
var fileToUpload = _configuration["FileToUpload"];
Console.WriteLine($"Uploading {fileToUpload} to the S3 bucket, {bucketName}.");
var success = await TransferMethods.UploadSingleFileAsync(transferUtil, bucketName, fileToUpload, localPath);
if (success)
{
Console.WriteLine($"Successfully uploaded the file, {fileToUpload} to {bucketName}.");
}
PressEnter();
// Upload a local directory to an S3 bucket.
DisplayTitle("Upload all files from a local directory");
Console.WriteLine("Upload all the files in a local folder to an S3 bucket.");
const string keyPrefix = "UploadFolder";
var uploadPath = $"{localPath}\\UploadFolder";
Console.WriteLine($"Uploading the files in {uploadPath} to {bucketName}");
DisplayTitle($"{uploadPath} files");
DisplayLocalFiles(uploadPath);
Console.WriteLine();
PressEnter();
success = await TransferMethods.UploadFullDirectoryAsync(transferUtil, bucketName, keyPrefix, uploadPath);
if (success)
{
Console.WriteLine($"Successfully uploaded the files in {uploadPath} to {bucketName}.");
Console.WriteLine($"{bucketName} currently contains the following files:");
await DisplayBucketFiles(client, bucketName, keyPrefix);
Console.WriteLine();
}
PressEnter();
// Download a single file from an S3 bucket.
DisplayTitle("Download a single file");
Console.WriteLine("Now we will download a single file from an S3 bucket.");
var keyName = _configuration["FileToDownload"];
Console.WriteLine($"Downloading {keyName} from {bucketName}.");
success = await TransferMethods.DownloadSingleFileAsync(transferUtil, bucketName, keyName, localPath);
if (success)
{
Console.WriteLine("$Successfully downloaded the file, {keyName} from {bucketName}.");
}
PressEnter();
// Download the contents of a directory from an S3 bucket.
DisplayTitle("Download the contents of an S3 bucket");
var s3Path = _configuration["S3Path"];
var downloadPath = $"{localPath}\\{s3Path}";
Console.WriteLine($"Downloading the contents of {bucketName}\\{s3Path}");
Console.WriteLine($"{bucketName}\\{s3Path} contains the following files:");
await DisplayBucketFiles(client, bucketName, s3Path);
Console.WriteLine();
success = await TransferMethods.DownloadS3DirectoryAsync(transferUtil, bucketName, s3Path, downloadPath);
if (success)
{
Console.WriteLine($"Downloaded the files in {bucketName} to {downloadPath}.");
Console.WriteLine($"{downloadPath} now contains the following files:");
DisplayLocalFiles(downloadPath);
}
Console.WriteLine("\nThe TransferUtility Basics application has completed.");
PressEnter();
// Displays the title for a section of the scenario.
static void DisplayTitle(string titleText)
{
var sepBar = new string('-', Console.WindowWidth);
Console.WriteLine(sepBar);
Console.WriteLine(CenterText(titleText));
Console.WriteLine(sepBar);
}
// Displays a description of the actions to be performed by the scenario.
static void DisplayInstructions()
{
var sepBar = new string('-', Console.WindowWidth);
DisplayTitle("Amazon S3 Transfer Utility Basics");
Console.WriteLine("This program shows how to use the Amazon S3 Transfer Utility.");
Console.WriteLine("It performs the following actions:");
Console.WriteLine("\t1. Upload a single object to an S3 bucket.");
Console.WriteLine("\t2. Upload an entire directory from the local computer to an\n\t S3 bucket.");
Console.WriteLine("\t3. Download a single object from an S3 bucket.");
Console.WriteLine("\t4. Download the objects in an S3 bucket to a local directory.");
Console.WriteLine($"\n{sepBar}");
}
// Pauses the scenario.
static void PressEnter()
{
Console.WriteLine("Press to continue.");
_ = Console.ReadLine();
Console.WriteLine("\n");
}
// Returns the string textToCenter, padded on the left with spaces
// that center the text on the console display.
static string CenterText(string textToCenter)
{
var centeredText = new StringBuilder();
var screenWidth = Console.WindowWidth;
centeredText.Append(new string(' ', (int)(screenWidth - textToCenter.Length) / 2));
centeredText.Append(textToCenter);
return centeredText.ToString();
}
// Displays a list of file names included in the specified path.
static void DisplayLocalFiles(string localPath)
{
var fileList = Directory.GetFiles(localPath);
if (fileList.Length > 0)
{
foreach (var fileName in fileList)
{
Console.WriteLine(fileName);
}
}
}
// Displays a list of the files in the specified S3 bucket and prefix.
static async Task DisplayBucketFiles(IAmazonS3 client, string bucketName, string s3Path)
{
ListObjectsV2Request request = new()
{
BucketName = bucketName,
Prefix = s3Path,
MaxKeys = 5,
};
var response = new ListObjectsV2Response();
do
{
response = await client.ListObjectsV2Async(request);
response.S3Objects
.ForEach(obj => Console.WriteLine($"{obj.Key}"));
// If the response is truncated, set the request ContinuationToken
// from the NextContinuationToken property of the response.
request.ContinuationToken = response.NextContinuationToken;
} while (response.IsTruncated);
}
```
Upload a single file.
```
///
/// Uploads a single file from the local computer to an S3 bucket.
///
/// The transfer initialized TransferUtility
/// object.
/// The name of the S3 bucket where the file
/// will be stored.
/// The name of the file to upload.
/// The local path where the file is stored.
/// A boolean value indicating the success of the action.
public static async Task UploadSingleFileAsync(
TransferUtility transferUtil,
string bucketName,
string fileName,
string localPath)
{
if (File.Exists($"{localPath}\\{fileName}"))
{
try
{
await transferUtil.UploadAsync(new TransferUtilityUploadRequest
{
BucketName = bucketName,
Key = fileName,
FilePath = $"{localPath}\\{fileName}",
});
return true;
}
catch (AmazonS3Exception s3Ex)
{
Console.WriteLine($"Could not upload {fileName} from {localPath} because:");
Console.WriteLine(s3Ex.Message);
return false;
}
}
else
{
Console.WriteLine($"{fileName} does not exist in {localPath}");
return false;
}
}
```
Upload an entire local directory.
```
///
/// Uploads all the files in a local directory to a directory in an S3
/// bucket.
///
/// The transfer initialized TransferUtility
/// object.
/// The name of the S3 bucket where the files
/// will be stored.
/// The key prefix is the S3 directory where
/// the files will be stored.
/// The local directory that contains the files
/// to be uploaded.
/// A Boolean value representing the success of the action.
public static async Task UploadFullDirectoryAsync(
TransferUtility transferUtil,
string bucketName,
string keyPrefix,
string localPath)
{
if (Directory.Exists(localPath))
{
try
{
await transferUtil.UploadDirectoryAsync(new TransferUtilityUploadDirectoryRequest
{
BucketName = bucketName,
KeyPrefix = keyPrefix,
Directory = localPath,
});
return true;
}
catch (AmazonS3Exception s3Ex)
{
Console.WriteLine($"Can't upload the contents of {localPath} because:");
Console.WriteLine(s3Ex?.Message);
return false;
}
}
else
{
Console.WriteLine($"The directory {localPath} does not exist.");
return false;
}
}
```
Download a single file.
```
///
/// Download a single file from an S3 bucket to the local computer.
///
/// The transfer initialized TransferUtility
/// object.
/// The name of the S3 bucket containing the
/// file to download.
/// The name of the file to download.
/// The path on the local computer where the
/// downloaded file will be saved.
/// A Boolean value indicating the results of the action.
public static async Task DownloadSingleFileAsync(
TransferUtility transferUtil,
string bucketName,
string keyName,
string localPath)
{
await transferUtil.DownloadAsync(new TransferUtilityDownloadRequest
{
BucketName = bucketName,
Key = keyName,
FilePath = $"{localPath}\\{keyName}",
});
return (File.Exists($"{localPath}\\{keyName}"));
}
```
Download contents of an S3 bucket.
```
///
/// Downloads the contents of a directory in an S3 bucket to a
/// directory on the local computer.
///
/// The transfer initialized TransferUtility
/// object.
/// The bucket containing the files to download.
/// The S3 directory where the files are located.
/// The local path to which the files will be
/// saved.
/// A Boolean value representing the success of the action.
public static async Task DownloadS3DirectoryAsync(
TransferUtility transferUtil,
string bucketName,
string s3Path,
string localPath)
{
int fileCount = 0;
// If the directory doesn't exist, it will be created.
if (Directory.Exists(s3Path))
{
var files = Directory.GetFiles(localPath);
fileCount = files.Length;
}
await transferUtil.DownloadDirectoryAsync(new TransferUtilityDownloadDirectoryRequest
{
BucketName = bucketName,
LocalDirectory = localPath,
S3Directory = s3Path,
});
if (Directory.Exists(localPath))
{
var files = Directory.GetFiles(localPath);
if (files.Length > fileCount)
{
return true;
}
// No change in the number of files. Assume
// the download failed.
return false;
}
// The local directory doesn't exist. No files
// were downloaded.
return false;
}
```
Track the progress of an upload using the TransferUtility.
```
using System;
using System.Threading.Tasks;
using Amazon.S3;
using Amazon.S3.Transfer;
///
/// This example shows how to track the progress of a multipart upload
/// using the Amazon Simple Storage Service (Amazon S3) TransferUtility to
/// upload to an Amazon S3 bucket.
///
public class TrackMPUUsingHighLevelAPI
{
public static async Task Main()
{
string bucketName = "amzn-s3-demo-bucket";
string keyName = "sample_pic.png";
string path = "filepath/directory/";
string filePath = $"{path}{keyName}";
// If the AWS Region defined for your default user is different
// from the Region where your Amazon S3 bucket is located,
// pass the Region name to the Amazon S3 client object's constructor.
// For example: RegionEndpoint.USWest2 or RegionEndpoint.USEast2.
IAmazonS3 client = new AmazonS3Client();
await TrackMPUAsync(client, bucketName, filePath, keyName);
}
///
/// Starts an Amazon S3 multipart upload and assigns an event handler to
/// track the progress of the upload.
///
/// The initialized Amazon S3 client object used to
/// perform the multipart upload.
/// The name of the bucket to which to upload
/// the file.
/// The path, including the file name of the
/// file to be uploaded to the Amazon S3 bucket.
/// The file name to be used in the
/// destination Amazon S3 bucket.
public static async Task TrackMPUAsync(
IAmazonS3 client,
string bucketName,
string filePath,
string keyName)
{
try
{
var fileTransferUtility = new TransferUtility(client);
// Use TransferUtilityUploadRequest to configure options.
// In this example we subscribe to an event.
var uploadRequest =
new TransferUtilityUploadRequest
{
BucketName = bucketName,
FilePath = filePath,
Key = keyName,
};
uploadRequest.UploadProgressEvent +=
new EventHandler(
UploadRequest_UploadPartProgressEvent);
await fileTransferUtility.UploadAsync(uploadRequest);
Console.WriteLine("Upload completed");
}
catch (AmazonS3Exception ex)
{
Console.WriteLine($"Error:: {ex.Message}");
}
}
///
/// Event handler to check the progress of the multipart upload.
///
/// The object that raised the event.
/// The object that contains multipart upload
/// information.
public static void UploadRequest_UploadPartProgressEvent(object sender, UploadProgressArgs e)
{
// Process event.
Console.WriteLine($"{e.TransferredBytes}/{e.TotalBytes}");
}
}
```
Upload an object with encryption.
```
using System;
using System.Collections.Generic;
using System.IO;
using System.Security.Cryptography;
using System.Threading.Tasks;
using Amazon.S3;
using Amazon.S3.Model;
///
/// Uses the Amazon Simple Storage Service (Amazon S3) low level API to
/// perform a multipart upload to an Amazon S3 bucket.
///
public class SSECLowLevelMPUcopyObject
{
public static async Task Main()
{
string existingBucketName = "amzn-s3-demo-bucket";
string sourceKeyName = "sample_file.txt";
string targetKeyName = "sample_file_copy.txt";
string filePath = $"sample\\{targetKeyName}";
// If the AWS Region defined for your default user is different
// from the Region where your Amazon S3 bucket is located,
// pass the Region name to the Amazon S3 client object's constructor.
// For example: RegionEndpoint.USEast1.
IAmazonS3 client = new AmazonS3Client();
// Create the encryption key.
var base64Key = CreateEncryptionKey();
await CreateSampleObjUsingClientEncryptionKeyAsync(
client,
existingBucketName,
sourceKeyName,
filePath,
base64Key);
}
///
/// Creates the encryption key to use with the multipart upload.
///
/// A string containing the base64-encoded key for encrypting
/// the multipart upload.
public static string CreateEncryptionKey()
{
Aes aesEncryption = Aes.Create();
aesEncryption.KeySize = 256;
aesEncryption.GenerateKey();
string base64Key = Convert.ToBase64String(aesEncryption.Key);
return base64Key;
}
///
/// Creates and uploads an object using a multipart upload.
///
/// The initialized Amazon S3 object used to
/// initialize and perform the multipart upload.
/// The name of the bucket to which
/// the object will be uploaded.
/// The source object name.
/// The location of the source object.
/// The encryption key to use with the upload.
public static async Task CreateSampleObjUsingClientEncryptionKeyAsync(
IAmazonS3 client,
string existingBucketName,
string sourceKeyName,
string filePath,
string base64Key)
{
List uploadResponses = new List();
InitiateMultipartUploadRequest initiateRequest = new InitiateMultipartUploadRequest
{
BucketName = existingBucketName,
Key = sourceKeyName,
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = base64Key,
};
InitiateMultipartUploadResponse initResponse =
await client.InitiateMultipartUploadAsync(initiateRequest);
long contentLength = new FileInfo(filePath).Length;
long partSize = 5 * (long)Math.Pow(2, 20); // 5 MB
try
{
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++)
{
UploadPartRequest uploadRequest = new UploadPartRequest
{
BucketName = existingBucketName,
Key = sourceKeyName,
UploadId = initResponse.UploadId,
PartNumber = i,
PartSize = partSize,
FilePosition = filePosition,
FilePath = filePath,
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = base64Key,
};
// Upload part and add response to our list.
uploadResponses.Add(await client.UploadPartAsync(uploadRequest));
filePosition += partSize;
}
CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest
{
BucketName = existingBucketName,
Key = sourceKeyName,
UploadId = initResponse.UploadId,
};
completeRequest.AddPartETags(uploadResponses);
CompleteMultipartUploadResponse completeUploadResponse =
await client.CompleteMultipartUploadAsync(completeRequest);
}
catch (Exception exception)
{
Console.WriteLine($"Exception occurred: {exception.Message}");
// If there was an error, abort the multipart upload.
AbortMultipartUploadRequest abortMPURequest = new AbortMultipartUploadRequest
{
BucketName = existingBucketName,
Key = sourceKeyName,
UploadId = initResponse.UploadId,
};
await client.AbortMultipartUploadAsync(abortMPURequest);
}
}
}
```
------
#### [ Go ]
**SDK for Go V2**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/gov2/s3#code-examples).
Create functions that use upload and download managers to break the data into parts and transfer them concurrently.
```
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"log"
"os"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/aws/smithy-go"
)
// BucketBasics encapsulates the Amazon Simple Storage Service (Amazon S3) actions
// used in the examples.
// It contains S3Client, an Amazon S3 service client that is used to perform bucket
// and object actions.
type BucketBasics struct {
S3Client *s3.Client
}
// UploadLargeObject uses an upload manager to upload data to an object in a bucket.
// The upload manager breaks large data into parts and uploads the parts concurrently.
func (basics BucketBasics) UploadLargeObject(ctx context.Context, bucketName string, objectKey string, largeObject []byte) error {
largeBuffer := bytes.NewReader(largeObject)
var partMiBs int64 = 10
uploader := manager.NewUploader(basics.S3Client, func(u *manager.Uploader) {
u.PartSize = partMiBs * 1024 * 1024
})
_, err := uploader.Upload(ctx, &s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
Body: largeBuffer,
})
if err != nil {
var apiErr smithy.APIError
if errors.As(err, &apiErr) && apiErr.ErrorCode() == "EntityTooLarge" {
log.Printf("Error while uploading object to %s. The object is too large.\n"+
"The maximum size for a multipart upload is 5TB.", bucketName)
} else {
log.Printf("Couldn't upload large object to %v:%v. Here's why: %v\n",
bucketName, objectKey, err)
}
} else {
err = s3.NewObjectExistsWaiter(basics.S3Client).Wait(
ctx, &s3.HeadObjectInput{Bucket: aws.String(bucketName), Key: aws.String(objectKey)}, time.Minute)
if err != nil {
log.Printf("Failed attempt to wait for object %s to exist.\n", objectKey)
}
}
return err
}
// DownloadLargeObject uses a download manager to download an object from a bucket.
// The download manager gets the data in parts and writes them to a buffer until all of
// the data has been downloaded.
func (basics BucketBasics) DownloadLargeObject(ctx context.Context, bucketName string, objectKey string) ([]byte, error) {
var partMiBs int64 = 10
downloader := manager.NewDownloader(basics.S3Client, func(d *manager.Downloader) {
d.PartSize = partMiBs * 1024 * 1024
})
buffer := manager.NewWriteAtBuffer([]byte{})
_, err := downloader.Download(ctx, buffer, &s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
if err != nil {
log.Printf("Couldn't download large object from %v:%v. Here's why: %v\n",
bucketName, objectKey, err)
}
return buffer.Bytes(), err
}
```
Run an interactive scenario that shows you how to use the upload and download managers in context.
```
import (
"context"
"crypto/rand"
"log"
"strings"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/awsdocs/aws-doc-sdk-examples/gov2/demotools"
"github.com/awsdocs/aws-doc-sdk-examples/gov2/s3/actions"
)
// RunLargeObjectScenario is an interactive example that shows you how to use Amazon
// Simple Storage Service (Amazon S3) to upload and download large objects.
//
// 1. Create a bucket.
// 3. Upload a large object to the bucket by using an upload manager.
// 5. Download a large object by using a download manager.
// 8. Delete all objects in the bucket.
// 9. Delete the bucket.
//
// This example creates an Amazon S3 service client from the specified sdkConfig so that
// you can replace it with a mocked or stubbed config for unit testing.
//
// It uses a questioner from the `demotools` package to get input during the example.
// This package can be found in the ..\..\demotools folder of this repo.
func RunLargeObjectScenario(ctx context.Context, sdkConfig aws.Config, questioner demotools.IQuestioner) {
defer func() {
if r := recover(); r != nil {
log.Println("Something went wrong with the demo.")
_, isMock := questioner.(*demotools.MockQuestioner)
if isMock || questioner.AskBool("Do you want to see the full error message (y/n)?", "y") {
log.Println(r)
}
}
}()
log.Println(strings.Repeat("-", 88))
log.Println("Welcome to the Amazon S3 large object demo.")
log.Println(strings.Repeat("-", 88))
s3Client := s3.NewFromConfig(sdkConfig)
bucketBasics := actions.BucketBasics{S3Client: s3Client}
bucketName := questioner.Ask("Let's create a bucket. Enter a name for your bucket:",
demotools.NotEmpty{})
bucketExists, err := bucketBasics.BucketExists(ctx, bucketName)
if err != nil {
panic(err)
}
if !bucketExists {
err = bucketBasics.CreateBucket(ctx, bucketName, sdkConfig.Region)
if err != nil {
panic(err)
} else {
log.Println("Bucket created.")
}
}
log.Println(strings.Repeat("-", 88))
mibs := 30
log.Printf("Let's create a slice of %v MiB of random bytes and upload it to your bucket. ", mibs)
questioner.Ask("Press Enter when you're ready.")
largeBytes := make([]byte, 1024*1024*mibs)
_, _ = rand.Read(largeBytes)
largeKey := "doc-example-large"
log.Println("Uploading...")
err = bucketBasics.UploadLargeObject(ctx, bucketName, largeKey, largeBytes)
if err != nil {
panic(err)
}
log.Printf("Uploaded %v MiB object as %v", mibs, largeKey)
log.Println(strings.Repeat("-", 88))
log.Printf("Let's download the %v MiB object.", mibs)
questioner.Ask("Press Enter when you're ready.")
log.Println("Downloading...")
largeDownload, err := bucketBasics.DownloadLargeObject(ctx, bucketName, largeKey)
if err != nil {
panic(err)
}
log.Printf("Downloaded %v bytes.", len(largeDownload))
log.Println(strings.Repeat("-", 88))
if questioner.AskBool("Do you want to delete your bucket and all of its "+
"contents? (y/n)", "y") {
log.Println("Deleting object.")
err = bucketBasics.DeleteObjects(ctx, bucketName, []string{largeKey})
if err != nil {
panic(err)
}
log.Println("Deleting bucket.")
err = bucketBasics.DeleteBucket(ctx, bucketName)
if err != nil {
panic(err)
}
} else {
log.Println("Okay. Don't forget to delete objects from your bucket to avoid charges.")
}
log.Println(strings.Repeat("-", 88))
log.Println("Thanks for watching!")
log.Println(strings.Repeat("-", 88))
}
```
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
Call functions that transfer files to and from an S3 bucket using the S3TransferManager.
```
public Integer downloadObjectsToDirectory(S3TransferManager transferManager,
URI destinationPathURI, String bucketName) {
DirectoryDownload directoryDownload = transferManager.downloadDirectory(DownloadDirectoryRequest.builder()
.destination(Paths.get(destinationPathURI))
.bucket(bucketName)
.build());
CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();
completedDirectoryDownload.failedTransfers()
.forEach(fail -> logger.warn("Object [{}] failed to transfer", fail.toString()));
return completedDirectoryDownload.failedTransfers().size();
}
```
Upload an entire local directory.
```
public Integer uploadDirectory(S3TransferManager transferManager,
URI sourceDirectory, String bucketName) {
DirectoryUpload directoryUpload = transferManager.uploadDirectory(UploadDirectoryRequest.builder()
.source(Paths.get(sourceDirectory))
.bucket(bucketName)
.build());
CompletedDirectoryUpload completedDirectoryUpload = directoryUpload.completionFuture().join();
completedDirectoryUpload.failedTransfers()
.forEach(fail -> logger.warn("Object [{}] failed to transfer", fail.toString()));
return completedDirectoryUpload.failedTransfers().size();
}
```
Upload a single file.
```
public String uploadFile(S3TransferManager transferManager, String bucketName,
String key, URI filePathURI) {
UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
.putObjectRequest(b -> b.bucket(bucketName).key(key))
.source(Paths.get(filePathURI))
.build();
FileUpload fileUpload = transferManager.uploadFile(uploadFileRequest);
CompletedFileUpload uploadResult = fileUpload.completionFuture().join();
return uploadResult.response().eTag();
}
```
The code examples use the following imports.
```
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload;
import software.amazon.awssdk.services.s3.model.CompletedPart;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import software.amazon.awssdk.services.s3.model.UploadPartRequest;
import software.amazon.awssdk.services.s3.model.UploadPartResponse;
import software.amazon.awssdk.services.s3.waiters.S3Waiter;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.FileUpload;
import software.amazon.awssdk.transfer.s3.model.UploadFileRequest;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.ByteBuffer;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
```
Use the [S3 Transfer Manager](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html) on top of the [AWS CRT-based S3 client](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html) to transparently perform a multipart upload when the size of the content exceeds a threshold. The default threshold size is 8 MB.
```
/**
* Uploads a file to an Amazon S3 bucket using the S3TransferManager.
*
* @param filePath the file path of the file to be uploaded
*/
public void multipartUploadWithTransferManager(String filePath) {
S3TransferManager transferManager = S3TransferManager.create();
UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
.putObjectRequest(b -> b
.bucket(bucketName)
.key(key))
.source(Paths.get(filePath))
.build();
FileUpload fileUpload = transferManager.uploadFile(uploadFileRequest);
fileUpload.completionFuture().join();
transferManager.close();
}
```
Use the [S3Client API](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Client.html) to perform a multipart upload.
```
/**
* Performs a multipart upload to Amazon S3 using the provided S3 client.
*
* @param filePath the path to the file to be uploaded
*/
public void multipartUploadWithS3Client(String filePath) {
// Initiate the multipart upload.
CreateMultipartUploadResponse createMultipartUploadResponse = s3Client.createMultipartUpload(b -> b
.bucket(bucketName)
.key(key));
String uploadId = createMultipartUploadResponse.uploadId();
// Upload the parts of the file.
int partNumber = 1;
List completedParts = new ArrayList<>();
ByteBuffer bb = ByteBuffer.allocate(1024 * 1024 * 5); // 5 MB byte buffer
try (RandomAccessFile file = new RandomAccessFile(filePath, "r")) {
long fileSize = file.length();
long position = 0;
while (position < fileSize) {
file.seek(position);
long read = file.getChannel().read(bb);
bb.flip(); // Swap position and limit before reading from the buffer.
UploadPartRequest uploadPartRequest = UploadPartRequest.builder()
.bucket(bucketName)
.key(key)
.uploadId(uploadId)
.partNumber(partNumber)
.build();
UploadPartResponse partResponse = s3Client.uploadPart(
uploadPartRequest,
RequestBody.fromByteBuffer(bb));
CompletedPart part = CompletedPart.builder()
.partNumber(partNumber)
.eTag(partResponse.eTag())
.build();
completedParts.add(part);
bb.clear();
position += read;
partNumber++;
}
} catch (IOException e) {
logger.error(e.getMessage());
}
// Complete the multipart upload.
s3Client.completeMultipartUpload(b -> b
.bucket(bucketName)
.key(key)
.uploadId(uploadId)
.multipartUpload(CompletedMultipartUpload.builder().parts(completedParts).build()));
}
```
Use the [S3AsyncClient API](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3AsyncClient.html) with multipart support enabled to perform a multipart upload.
```
/**
* Uploads a file to an S3 bucket using the S3AsyncClient and enabling multipart support.
*
* @param filePath the local file path of the file to be uploaded
*/
public void multipartUploadWithS3AsyncClient(String filePath) {
// Enable multipart support.
S3AsyncClient s3AsyncClient = S3AsyncClient.builder()
.multipartEnabled(true)
.build();
CompletableFuture response = s3AsyncClient.putObject(b -> b
.bucket(bucketName)
.key(key),
Paths.get(filePath));
response.join();
logger.info("File uploaded in multiple 8 MiB parts using S3AsyncClient.");
}
```
------
#### [ JavaScript ]
**SDK for JavaScript (v3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javascriptv3/example_code/s3#code-examples).
Upload a large file.
```
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import {
ProgressBar,
logger,
} from "@aws-doc-sdk-examples/lib/utils/util-log.js";
const twentyFiveMB = 25 * 1024 * 1024;
export const createString = (size = twentyFiveMB) => {
return "x".repeat(size);
};
/**
* Create a 25MB file and upload it in parts to the specified
* Amazon S3 bucket.
* @param {{ bucketName: string, key: string }}
*/
export const main = async ({ bucketName, key }) => {
const str = createString();
const buffer = Buffer.from(str, "utf8");
const progressBar = new ProgressBar({
description: `Uploading "${key}" to "${bucketName}"`,
barLength: 30,
});
try {
const upload = new Upload({
client: new S3Client({}),
params: {
Bucket: bucketName,
Key: key,
Body: buffer,
},
});
upload.on("httpUploadProgress", ({ loaded, total }) => {
progressBar.update({ current: loaded, total });
});
await upload.done();
} catch (caught) {
if (caught instanceof Error && caught.name === "AbortError") {
logger.error(`Multipart upload was aborted. ${caught.message}`);
} else {
throw caught;
}
}
};
```
Download a large file.
```
import { fileURLToPath } from "node:url";
import { GetObjectCommand, NoSuchKey, S3Client } from "@aws-sdk/client-s3";
import { createWriteStream, rmSync } from "node:fs";
const s3Client = new S3Client({});
const oneMB = 1024 * 1024;
export const getObjectRange = ({ bucket, key, start, end }) => {
const command = new GetObjectCommand({
Bucket: bucket,
Key: key,
Range: `bytes=${start}-${end}`,
});
return s3Client.send(command);
};
/**
* @param {string | undefined} contentRange
*/
export const getRangeAndLength = (contentRange) => {
const [range, length] = contentRange.split("/");
const [start, end] = range.split("-");
return {
start: Number.parseInt(start),
end: Number.parseInt(end),
length: Number.parseInt(length),
};
};
export const isComplete = ({ end, length }) => end === length - 1;
const downloadInChunks = async ({ bucket, key }) => {
const writeStream = createWriteStream(
fileURLToPath(new URL(`./${key}`, import.meta.url)),
).on("error", (err) => console.error(err));
let rangeAndLength = { start: -1, end: -1, length: -1 };
while (!isComplete(rangeAndLength)) {
const { end } = rangeAndLength;
const nextRange = { start: end + 1, end: end + oneMB };
const { ContentRange, Body } = await getObjectRange({
bucket,
key,
...nextRange,
});
console.log(`Downloaded bytes ${nextRange.start} to ${nextRange.end}`);
writeStream.write(await Body.transformToByteArray());
rangeAndLength = getRangeAndLength(ContentRange);
}
};
/**
* Download a large object from and Amazon S3 bucket.
*
* When downloading a large file, you might want to break it down into
* smaller pieces. Amazon S3 accepts a Range header to specify the start
* and end of the byte range to be downloaded.
*
* @param {{ bucketName: string, key: string }}
*/
export const main = async ({ bucketName, key }) => {
try {
await downloadInChunks({
bucket: bucketName,
key: key,
});
} catch (caught) {
if (caught instanceof NoSuchKey) {
console.error(`Failed to download object. No such key "${key}".`);
rmSync(key);
}
}
};
```
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/s3/file_transfer#code-examples).
Create functions that transfer files using several of the available transfer manager settings. Use a callback class to write callback progress during file transfer.
```
import sys
import threading
import boto3
from boto3.s3.transfer import TransferConfig
MB = 1024 * 1024
s3 = boto3.resource("s3")
class TransferCallback:
"""
Handle callbacks from the transfer manager.
The transfer manager periodically calls the __call__ method throughout
the upload and download process so that it can take action, such as
displaying progress to the user and collecting data about the transfer.
"""
def __init__(self, target_size):
self._target_size = target_size
self._total_transferred = 0
self._lock = threading.Lock()
self.thread_info = {}
def __call__(self, bytes_transferred):
"""
The callback method that is called by the transfer manager.
Display progress during file transfer and collect per-thread transfer
data. This method can be called by multiple threads, so shared instance
data is protected by a thread lock.
"""
thread = threading.current_thread()
with self._lock:
self._total_transferred += bytes_transferred
if thread.ident not in self.thread_info.keys():
self.thread_info[thread.ident] = bytes_transferred
else:
self.thread_info[thread.ident] += bytes_transferred
target = self._target_size * MB
sys.stdout.write(
f"\r{self._total_transferred} of {target} transferred "
f"({(self._total_transferred / target) * 100:.2f}%)."
)
sys.stdout.flush()
def upload_with_default_configuration(
local_file_path, bucket_name, object_key, file_size_mb
):
"""
Upload a file from a local folder to an Amazon S3 bucket, using the default
configuration.
"""
transfer_callback = TransferCallback(file_size_mb)
s3.Bucket(bucket_name).upload_file(
local_file_path, object_key, Callback=transfer_callback
)
return transfer_callback.thread_info
def upload_with_chunksize_and_meta(
local_file_path, bucket_name, object_key, file_size_mb, metadata=None
):
"""
Upload a file from a local folder to an Amazon S3 bucket, setting a
multipart chunk size and adding metadata to the Amazon S3 object.
The multipart chunk size controls the size of the chunks of data that are
sent in the request. A smaller chunk size typically results in the transfer
manager using more threads for the upload.
The metadata is a set of key-value pairs that are stored with the object
in Amazon S3.
"""
transfer_callback = TransferCallback(file_size_mb)
config = TransferConfig(multipart_chunksize=1 * MB)
extra_args = {"Metadata": metadata} if metadata else None
s3.Bucket(bucket_name).upload_file(
local_file_path,
object_key,
Config=config,
ExtraArgs=extra_args,
Callback=transfer_callback,
)
return transfer_callback.thread_info
def upload_with_high_threshold(local_file_path, bucket_name, object_key, file_size_mb):
"""
Upload a file from a local folder to an Amazon S3 bucket, setting a
multipart threshold larger than the size of the file.
Setting a multipart threshold larger than the size of the file results
in the transfer manager sending the file as a standard upload instead of
a multipart upload.
"""
transfer_callback = TransferCallback(file_size_mb)
config = TransferConfig(multipart_threshold=file_size_mb * 2 * MB)
s3.Bucket(bucket_name).upload_file(
local_file_path, object_key, Config=config, Callback=transfer_callback
)
return transfer_callback.thread_info
def upload_with_sse(
local_file_path, bucket_name, object_key, file_size_mb, sse_key=None
):
"""
Upload a file from a local folder to an Amazon S3 bucket, adding server-side
encryption with customer-provided encryption keys to the object.
When this kind of encryption is specified, Amazon S3 encrypts the object
at rest and allows downloads only when the expected encryption key is
provided in the download request.
"""
transfer_callback = TransferCallback(file_size_mb)
if sse_key:
extra_args = {"SSECustomerAlgorithm": "AES256", "SSECustomerKey": sse_key}
else:
extra_args = None
s3.Bucket(bucket_name).upload_file(
local_file_path, object_key, ExtraArgs=extra_args, Callback=transfer_callback
)
return transfer_callback.thread_info
def download_with_default_configuration(
bucket_name, object_key, download_file_path, file_size_mb
):
"""
Download a file from an Amazon S3 bucket to a local folder, using the
default configuration.
"""
transfer_callback = TransferCallback(file_size_mb)
s3.Bucket(bucket_name).Object(object_key).download_file(
download_file_path, Callback=transfer_callback
)
return transfer_callback.thread_info
def download_with_single_thread(
bucket_name, object_key, download_file_path, file_size_mb
):
"""
Download a file from an Amazon S3 bucket to a local folder, using a
single thread.
"""
transfer_callback = TransferCallback(file_size_mb)
config = TransferConfig(use_threads=False)
s3.Bucket(bucket_name).Object(object_key).download_file(
download_file_path, Config=config, Callback=transfer_callback
)
return transfer_callback.thread_info
def download_with_high_threshold(
bucket_name, object_key, download_file_path, file_size_mb
):
"""
Download a file from an Amazon S3 bucket to a local folder, setting a
multipart threshold larger than the size of the file.
Setting a multipart threshold larger than the size of the file results
in the transfer manager sending the file as a standard download instead
of a multipart download.
"""
transfer_callback = TransferCallback(file_size_mb)
config = TransferConfig(multipart_threshold=file_size_mb * 2 * MB)
s3.Bucket(bucket_name).Object(object_key).download_file(
download_file_path, Config=config, Callback=transfer_callback
)
return transfer_callback.thread_info
def download_with_sse(
bucket_name, object_key, download_file_path, file_size_mb, sse_key
):
"""
Download a file from an Amazon S3 bucket to a local folder, adding a
customer-provided encryption key to the request.
When this kind of encryption is specified, Amazon S3 encrypts the object
at rest and allows downloads only when the expected encryption key is
provided in the download request.
"""
transfer_callback = TransferCallback(file_size_mb)
if sse_key:
extra_args = {"SSECustomerAlgorithm": "AES256", "SSECustomerKey": sse_key}
else:
extra_args = None
s3.Bucket(bucket_name).Object(object_key).download_file(
download_file_path, ExtraArgs=extra_args, Callback=transfer_callback
)
return transfer_callback.thread_info
```
Demonstrate the transfer manager functions and report results.
```
import hashlib
import os
import platform
import shutil
import time
import boto3
from boto3.s3.transfer import TransferConfig
from botocore.exceptions import ClientError
from botocore.exceptions import ParamValidationError
from botocore.exceptions import NoCredentialsError
import file_transfer
MB = 1024 * 1024
# These configuration attributes affect both uploads and downloads.
CONFIG_ATTRS = (
"multipart_threshold",
"multipart_chunksize",
"max_concurrency",
"use_threads",
)
# These configuration attributes affect only downloads.
DOWNLOAD_CONFIG_ATTRS = ("max_io_queue", "io_chunksize", "num_download_attempts")
class TransferDemoManager:
"""
Manages the demonstration. Collects user input from a command line, reports
transfer results, maintains a list of artifacts created during the
demonstration, and cleans them up after the demonstration is completed.
"""
def __init__(self):
self._s3 = boto3.resource("s3")
self._chore_list = []
self._create_file_cmd = None
self._size_multiplier = 0
self.file_size_mb = 30
self.demo_folder = None
self.demo_bucket = None
self._setup_platform_specific()
self._terminal_width = shutil.get_terminal_size(fallback=(80, 80))[0]
def collect_user_info(self):
"""
Collect local folder and Amazon S3 bucket name from the user. These
locations are used to store files during the demonstration.
"""
while not self.demo_folder:
self.demo_folder = input(
"Which file folder do you want to use to store " "demonstration files? "
)
if not os.path.isdir(self.demo_folder):
print(f"{self.demo_folder} isn't a folder!")
self.demo_folder = None
while not self.demo_bucket:
self.demo_bucket = input(
"Which Amazon S3 bucket do you want to use to store "
"demonstration files? "
)
try:
self._s3.meta.client.head_bucket(Bucket=self.demo_bucket)
except ParamValidationError as err:
print(err)
self.demo_bucket = None
except ClientError as err:
print(err)
print(
f"Either {self.demo_bucket} doesn't exist or you don't "
f"have access to it."
)
self.demo_bucket = None
def demo(
self, question, upload_func, download_func, upload_args=None, download_args=None
):
"""Run a demonstration.
Ask the user if they want to run this specific demonstration.
If they say yes, create a file on the local path, upload it
using the specified upload function, then download it using the
specified download function.
"""
if download_args is None:
download_args = {}
if upload_args is None:
upload_args = {}
question = question.format(self.file_size_mb)
answer = input(f"{question} (y/n)")
if answer.lower() == "y":
local_file_path, object_key, download_file_path = self._create_demo_file()
file_transfer.TransferConfig = self._config_wrapper(
TransferConfig, CONFIG_ATTRS
)
self._report_transfer_params(
"Uploading", local_file_path, object_key, **upload_args
)
start_time = time.perf_counter()
thread_info = upload_func(
local_file_path,
self.demo_bucket,
object_key,
self.file_size_mb,
**upload_args,
)
end_time = time.perf_counter()
self._report_transfer_result(thread_info, end_time - start_time)
file_transfer.TransferConfig = self._config_wrapper(
TransferConfig, CONFIG_ATTRS + DOWNLOAD_CONFIG_ATTRS
)
self._report_transfer_params(
"Downloading", object_key, download_file_path, **download_args
)
start_time = time.perf_counter()
thread_info = download_func(
self.demo_bucket,
object_key,
download_file_path,
self.file_size_mb,
**download_args,
)
end_time = time.perf_counter()
self._report_transfer_result(thread_info, end_time - start_time)
def last_name_set(self):
"""Get the name set used for the last demo."""
return self._chore_list[-1]
def cleanup(self):
"""
Remove files from the demo folder, and uploaded objects from the
Amazon S3 bucket.
"""
print("-" * self._terminal_width)
for local_file_path, s3_object_key, downloaded_file_path in self._chore_list:
print(f"Removing {local_file_path}")
try:
os.remove(local_file_path)
except FileNotFoundError as err:
print(err)
print(f"Removing {downloaded_file_path}")
try:
os.remove(downloaded_file_path)
except FileNotFoundError as err:
print(err)
if self.demo_bucket:
print(f"Removing {self.demo_bucket}:{s3_object_key}")
try:
self._s3.Bucket(self.demo_bucket).Object(s3_object_key).delete()
except ClientError as err:
print(err)
def _setup_platform_specific(self):
"""Set up platform-specific command used to create a large file."""
if platform.system() == "Windows":
self._create_file_cmd = "fsutil file createnew {} {}"
self._size_multiplier = MB
elif platform.system() == "Linux" or platform.system() == "Darwin":
self._create_file_cmd = f"dd if=/dev/urandom of={{}} " f"bs={MB} count={{}}"
self._size_multiplier = 1
else:
raise EnvironmentError(
f"Demo of platform {platform.system()} isn't supported."
)
def _create_demo_file(self):
"""
Create a file in the demo folder specified by the user. Store the local
path, object name, and download path for later cleanup.
Only the local file is created by this method. The Amazon S3 object and
download file are created later during the demonstration.
Returns:
A tuple that contains the local file path, object name, and download
file path.
"""
file_name_template = "TestFile{}-{}.demo"
local_suffix = "local"
object_suffix = "s3object"
download_suffix = "downloaded"
file_tag = len(self._chore_list) + 1
local_file_path = os.path.join(
self.demo_folder, file_name_template.format(file_tag, local_suffix)
)
s3_object_key = file_name_template.format(file_tag, object_suffix)
downloaded_file_path = os.path.join(
self.demo_folder, file_name_template.format(file_tag, download_suffix)
)
filled_cmd = self._create_file_cmd.format(
local_file_path, self.file_size_mb * self._size_multiplier
)
print(
f"Creating file of size {self.file_size_mb} MB "
f"in {self.demo_folder} by running:"
)
print(f"{'':4}{filled_cmd}")
os.system(filled_cmd)
chore = (local_file_path, s3_object_key, downloaded_file_path)
self._chore_list.append(chore)
return chore
def _report_transfer_params(self, verb, source_name, dest_name, **kwargs):
"""Report configuration and extra arguments used for a file transfer."""
print("-" * self._terminal_width)
print(f"{verb} {source_name} ({self.file_size_mb} MB) to {dest_name}")
if kwargs:
print("With extra args:")
for arg, value in kwargs.items():
print(f'{"":4}{arg:<20}: {value}')
@staticmethod
def ask_user(question):
"""
Ask the user a yes or no question.
Returns:
True when the user answers 'y' or 'Y'; otherwise, False.
"""
answer = input(f"{question} (y/n) ")
return answer.lower() == "y"
@staticmethod
def _config_wrapper(func, config_attrs):
def wrapper(*args, **kwargs):
config = func(*args, **kwargs)
print("With configuration:")
for attr in config_attrs:
print(f'{"":4}{attr:<20}: {getattr(config, attr)}')
return config
return wrapper
@staticmethod
def _report_transfer_result(thread_info, elapsed):
"""Report the result of a transfer, including per-thread data."""
print(f"\nUsed {len(thread_info)} threads.")
for ident, byte_count in thread_info.items():
print(f"{'':4}Thread {ident} copied {byte_count} bytes.")
print(f"Your transfer took {elapsed:.2f} seconds.")
def main():
"""
Run the demonstration script for s3_file_transfer.
"""
demo_manager = TransferDemoManager()
demo_manager.collect_user_info()
# Upload and download with default configuration. Because the file is 30 MB
# and the default multipart_threshold is 8 MB, both upload and download are
# multipart transfers.
demo_manager.demo(
"Do you want to upload and download a {} MB file "
"using the default configuration?",
file_transfer.upload_with_default_configuration,
file_transfer.download_with_default_configuration,
)
# Upload and download with multipart_threshold set higher than the size of
# the file. This causes the transfer manager to use standard transfers
# instead of multipart transfers.
demo_manager.demo(
"Do you want to upload and download a {} MB file "
"as a standard (not multipart) transfer?",
file_transfer.upload_with_high_threshold,
file_transfer.download_with_high_threshold,
)
# Upload with specific chunk size and additional metadata.
# Download with a single thread.
demo_manager.demo(
"Do you want to upload a {} MB file with a smaller chunk size and "
"then download the same file using a single thread?",
file_transfer.upload_with_chunksize_and_meta,
file_transfer.download_with_single_thread,
upload_args={
"metadata": {
"upload_type": "chunky",
"favorite_color": "aqua",
"size": "medium",
}
},
)
# Upload using server-side encryption with customer-provided
# encryption keys.
# Generate a 256-bit key from a passphrase.
sse_key = hashlib.sha256("demo_passphrase".encode("utf-8")).digest()
demo_manager.demo(
"Do you want to upload and download a {} MB file using "
"server-side encryption?",
file_transfer.upload_with_sse,
file_transfer.download_with_sse,
upload_args={"sse_key": sse_key},
download_args={"sse_key": sse_key},
)
# Download without specifying an encryption key to show that the
# encryption key must be included to download an encrypted object.
if demo_manager.ask_user(
"Do you want to try to download the encrypted "
"object without sending the required key?"
):
try:
_, object_key, download_file_path = demo_manager.last_name_set()
file_transfer.download_with_default_configuration(
demo_manager.demo_bucket,
object_key,
download_file_path,
demo_manager.file_size_mb,
)
except ClientError as err:
print(
"Got expected error when trying to download an encrypted "
"object without specifying encryption info:"
)
print(f"{'':4}{err}")
# Remove all created and downloaded files, remove all objects from
# S3 storage.
if demo_manager.ask_user(
"Demonstration complete. Do you want to remove local files " "and S3 objects?"
):
demo_manager.cleanup()
if __name__ == "__main__":
try:
main()
except NoCredentialsError as error:
print(error)
print(
"To run this example, you must have valid credentials in "
"a shared credential file or set in environment variables."
)
```
------
#### [ Rust ]
**SDK for Rust**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/rustv1/examples/s3#code-examples).
```
use std::fs::File;
use std::io::prelude::*;
use std::path::Path;
use aws_config::meta::region::RegionProviderChain;
use aws_sdk_s3::error::DisplayErrorContext;
use aws_sdk_s3::operation::{
create_multipart_upload::CreateMultipartUploadOutput, get_object::GetObjectOutput,
};
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use aws_sdk_s3::{config::Region, Client as S3Client};
use aws_smithy_types::byte_stream::{ByteStream, Length};
use rand::distributions::Alphanumeric;
use rand::{thread_rng, Rng};
use s3_code_examples::error::S3ExampleError;
use std::process;
use uuid::Uuid;
//In bytes, minimum chunk size of 5MB. Increase CHUNK_SIZE to send larger chunks.
const CHUNK_SIZE: u64 = 1024 * 1024 * 5;
const MAX_CHUNKS: u64 = 10000;
#[tokio::main]
pub async fn main() {
if let Err(err) = run_example().await {
eprintln!("Error: {}", DisplayErrorContext(err));
process::exit(1);
}
}
async fn run_example() -> Result<(), S3ExampleError> {
let shared_config = aws_config::load_from_env().await;
let client = S3Client::new(&shared_config);
let bucket_name = format!("amzn-s3-demo-bucket-{}", Uuid::new_v4());
let region_provider = RegionProviderChain::first_try(Region::new("us-west-2"));
let region = region_provider.region().await.unwrap();
s3_code_examples::create_bucket(&client, &bucket_name, ®ion).await?;
let key = "sample.txt".to_string();
// Create a multipart upload. Use UploadPart and CompleteMultipartUpload to
// upload the file.
let multipart_upload_res: CreateMultipartUploadOutput = client
.create_multipart_upload()
.bucket(&bucket_name)
.key(&key)
.send()
.await?;
let upload_id = multipart_upload_res.upload_id().ok_or(S3ExampleError::new(
"Missing upload_id after CreateMultipartUpload",
))?;
//Create a file of random characters for the upload.
let mut file = File::create(&key).expect("Could not create sample file.");
// Loop until the file is 5 chunks.
while file.metadata().unwrap().len() <= CHUNK_SIZE * 4 {
let rand_string: String = thread_rng()
.sample_iter(&Alphanumeric)
.take(256)
.map(char::from)
.collect();
let return_string: String = "\n".to_string();
file.write_all(rand_string.as_ref())
.expect("Error writing to file.");
file.write_all(return_string.as_ref())
.expect("Error writing to file.");
}
let path = Path::new(&key);
let file_size = tokio::fs::metadata(path)
.await
.expect("it exists I swear")
.len();
let mut chunk_count = (file_size / CHUNK_SIZE) + 1;
let mut size_of_last_chunk = file_size % CHUNK_SIZE;
if size_of_last_chunk == 0 {
size_of_last_chunk = CHUNK_SIZE;
chunk_count -= 1;
}
if file_size == 0 {
return Err(S3ExampleError::new("Bad file size."));
}
if chunk_count > MAX_CHUNKS {
return Err(S3ExampleError::new(
"Too many chunks! Try increasing your chunk size.",
));
}
let mut upload_parts: Vec = Vec::new();
for chunk_index in 0..chunk_count {
let this_chunk = if chunk_count - 1 == chunk_index {
size_of_last_chunk
} else {
CHUNK_SIZE
};
let stream = ByteStream::read_from()
.path(path)
.offset(chunk_index * CHUNK_SIZE)
.length(Length::Exact(this_chunk))
.build()
.await
.unwrap();
// Chunk index needs to start at 0, but part numbers start at 1.
let part_number = (chunk_index as i32) + 1;
let upload_part_res = client
.upload_part()
.key(&key)
.bucket(&bucket_name)
.upload_id(upload_id)
.body(stream)
.part_number(part_number)
.send()
.await?;
upload_parts.push(
CompletedPart::builder()
.e_tag(upload_part_res.e_tag.unwrap_or_default())
.part_number(part_number)
.build(),
);
}
// upload_parts: Vec
let completed_multipart_upload: CompletedMultipartUpload = CompletedMultipartUpload::builder()
.set_parts(Some(upload_parts))
.build();
let _complete_multipart_upload_res = client
.complete_multipart_upload()
.bucket(&bucket_name)
.key(&key)
.multipart_upload(completed_multipart_upload)
.upload_id(upload_id)
.send()
.await?;
let data: GetObjectOutput =
s3_code_examples::download_object(&client, &bucket_name, &key).await?;
let data_length: u64 = data
.content_length()
.unwrap_or_default()
.try_into()
.unwrap();
if file.metadata().unwrap().len() == data_length {
println!("Data lengths match.");
} else {
println!("The data was not the same size!");
}
s3_code_examples::clear_bucket(&client, &bucket_name)
.await
.expect("Error emptying bucket.");
s3_code_examples::delete_bucket(&client, &bucket_name)
.await
.expect("Error deleting bucket.");
Ok(())
}
```
------
# Upload a stream of unknown size to an Amazon S3 object using an AWS SDK
The following code examples show how to upload a stream of unknown size to an Amazon S3 object.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
Use the [AWS CRT-based S3 Client](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html).
```
import com.example.s3.util.AsyncExampleUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class PutObjectFromStreamAsync {
private static final Logger logger = LoggerFactory.getLogger(PutObjectFromStreamAsync.class);
public static void main(String[] args) {
String bucketName = "amzn-s3-demo-bucket-" + UUID.randomUUID(); // Change bucket name.
String key = UUID.randomUUID().toString();
AsyncExampleUtils.createBucket(bucketName);
try {
PutObjectFromStreamAsync example = new PutObjectFromStreamAsync();
S3AsyncClient s3AsyncClientCrt = S3AsyncClient.crtCreate();
PutObjectResponse putObjectResponse = example.putObjectFromStreamCrt(s3AsyncClientCrt, bucketName, key);
logger.info("Object {} etag: {}", key, putObjectResponse.eTag());
logger.info("Object {} uploaded to bucket {}.", key, bucketName);
} catch (SdkException e) {
logger.error(e.getMessage(), e);
} finally {
AsyncExampleUtils.deleteObject(bucketName, key);
AsyncExampleUtils.deleteBucket(bucketName);
}
}
/**
* @param s33CrtAsyncClient - To upload content from a stream of unknown size, use can the AWS CRT-based S3 client.
* @param bucketName - The name of the bucket.
* @param key - The name of the object.
* @return software.amazon.awssdk.services.s3.model.PutObjectResponse - Returns metadata pertaining to the put object operation.
*/
public PutObjectResponse putObjectFromStreamCrt(S3AsyncClient s33CrtAsyncClient, String bucketName, String key) {
// AsyncExampleUtils.randomString() returns a random string up to 100 characters.
String randomString = AsyncExampleUtils.randomString();
logger.info("random string to upload: {}: length={}", randomString, randomString.length());
InputStream inputStream = new ByteArrayInputStream(randomString.getBytes());
// Executor required to handle reading from the InputStream on a separate thread so the main upload is not blocked.
ExecutorService executor = Executors.newSingleThreadExecutor();
// Specify `null` for the content length when you don't know the content length.
AsyncRequestBody body = AsyncRequestBody.fromInputStream(inputStream, null, executor);
CompletableFuture responseFuture =
s33CrtAsyncClient.putObject(r -> r.bucket(bucketName).key(key), body);
PutObjectResponse response = responseFuture.join(); // Wait for the response.
logger.info("Object {} uploaded to bucket {}.", key, bucketName);
executor.shutdown();
return response;
}
}
```
Use the standard [ asynchronous S3 client with multipart upload enabled](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/s3-async-client-multipart.html#s3-async-client-mp-on).
```
import com.example.s3.util.AsyncExampleUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class PutObjectFromStreamAsyncMp {
private static final Logger logger = LoggerFactory.getLogger(PutObjectFromStreamAsyncMp.class);
public static void main(String[] args) {
String bucketName = "amzn-s3-demo-bucket-" + UUID.randomUUID(); // Change bucket name.
String key = UUID.randomUUID().toString();
AsyncExampleUtils.createBucket(bucketName);
try {
PutObjectFromStreamAsyncMp example = new PutObjectFromStreamAsyncMp();
S3AsyncClient s3AsyncClientMp = S3AsyncClient.builder().multipartEnabled(true).build();
PutObjectResponse putObjectResponse = example.putObjectFromStreamMp(s3AsyncClientMp, bucketName, key);
logger.info("Object {} etag: {}", key, putObjectResponse.eTag());
logger.info("Object {} uploaded to bucket {}.", key, bucketName);
} catch (SdkException e) {
logger.error(e.getMessage(), e);
} finally {
AsyncExampleUtils.deleteObject(bucketName, key);
AsyncExampleUtils.deleteBucket(bucketName);
}
}
/**
* @param s3AsyncClientMp - To upload content from a stream of unknown size, use can the S3 asynchronous client with multipart enabled.
* @param bucketName - The name of the bucket.
* @param key - The name of the object.
* @return software.amazon.awssdk.services.s3.model.PutObjectResponse - Returns metadata pertaining to the put object operation.
*/
public PutObjectResponse putObjectFromStreamMp(S3AsyncClient s3AsyncClientMp, String bucketName, String key) {
// AsyncExampleUtils.randomString() returns a random string up to 100 characters.
String randomString = AsyncExampleUtils.randomString();
logger.info("random string to upload: {}: length={}", randomString, randomString.length());
InputStream inputStream = new ByteArrayInputStream(randomString.getBytes());
// Executor required to handle reading from the InputStream on a separate thread so the main upload is not blocked.
ExecutorService executor = Executors.newSingleThreadExecutor();
// Specify `null` for the content length when you don't know the content length.
AsyncRequestBody body = AsyncRequestBody.fromInputStream(inputStream, null, executor);
CompletableFuture responseFuture =
s3AsyncClientMp.putObject(r -> r.bucket(bucketName).key(key), body);
PutObjectResponse response = responseFuture.join(); // Wait for the response.
logger.info("Object {} uploaded to bucket {}.", key, bucketName);
executor.shutdown();
return response;
}
}
```
Use the [Amazon S3 Transfer Manager](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html).
```
import com.example.s3.util.AsyncExampleUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.CompletedUpload;
import software.amazon.awssdk.transfer.s3.model.Upload;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class UploadStream {
private static final Logger logger = LoggerFactory.getLogger(UploadStream.class);
public static void main(String[] args) {
String bucketName = "amzn-s3-demo-bucket" + UUID.randomUUID();
String key = UUID.randomUUID().toString();
AsyncExampleUtils.createBucket(bucketName);
try {
UploadStream example = new UploadStream();
CompletedUpload completedUpload = example.uploadStream(S3TransferManager.create(), bucketName, key);
logger.info("Object {} etag: {}", key, completedUpload.response().eTag());
logger.info("Object {} uploaded to bucket {}.", key, bucketName);
} catch (SdkException e) {
logger.error(e.getMessage(), e);
} finally {
AsyncExampleUtils.deleteObject(bucketName, key);
AsyncExampleUtils.deleteBucket(bucketName);
}
}
/**
* @param transferManager - To upload content from a stream of unknown size, you can use the S3TransferManager based on the AWS CRT-based S3 client.
* @param bucketName - The name of the bucket.
* @param key - The name of the object.
* @return - software.amazon.awssdk.transfer.s3.model.CompletedUpload - The result of the completed upload.
*/
public CompletedUpload uploadStream(S3TransferManager transferManager, String bucketName, String key) {
// AsyncExampleUtils.randomString() returns a random string up to 100 characters.
String randomString = AsyncExampleUtils.randomString();
logger.info("random string to upload: {}: length={}", randomString, randomString.length());
InputStream inputStream = new ByteArrayInputStream(randomString.getBytes());
// Executor required to handle reading from the InputStream on a separate thread so the main upload is not blocked.
ExecutorService executor = Executors.newSingleThreadExecutor();
// Specify `null` for the content length when you don't know the content length.
AsyncRequestBody body = AsyncRequestBody.fromInputStream(inputStream, null, executor);
Upload upload = transferManager.upload(builder -> builder
.requestBody(body)
.putObjectRequest(req -> req.bucket(bucketName).key(key))
.build());
CompletedUpload completedUpload = upload.completionFuture().join();
executor.shutdown();
return completedUpload;
}
}
```
------
#### [ Swift ]
**SDK for Swift**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/swift/example_code/s3/binary-streaming#code-examples).
```
import ArgumentParser
import AWSClientRuntime
import AWSS3
import Foundation
import Smithy
import SmithyHTTPAPI
import SmithyStreams
/// Upload a file to the specified bucket.
///
/// - Parameters:
/// - bucket: The Amazon S3 bucket name to store the file into.
/// - key: The name (or path) of the file to upload to in the `bucket`.
/// - sourcePath: The pathname on the local filesystem of the file to
/// upload.
func uploadFile(sourcePath: String, bucket: String, key: String?) async throws {
let fileURL: URL = URL(fileURLWithPath: sourcePath)
let fileName: String
// If no key was provided, use the last component of the filename.
if key == nil {
fileName = fileURL.lastPathComponent
} else {
fileName = key!
}
let s3Client = try await S3Client()
// Create a FileHandle for the source file.
let fileHandle = FileHandle(forReadingAtPath: sourcePath)
guard let fileHandle = fileHandle else {
throw TransferError.readError
}
// Create a byte stream to retrieve the file's contents. This uses the
// Smithy FileStream and ByteStream types.
let stream = FileStream(fileHandle: fileHandle)
let body = ByteStream.stream(stream)
// Create a `PutObjectInput` with the ByteStream as the body of the
// request's data. The AWS SDK for Swift will handle sending the
// entire file in chunks, regardless of its size.
let putInput = PutObjectInput(
body: body,
bucket: bucket,
key: fileName
)
do {
_ = try await s3Client.putObject(input: putInput)
} catch {
throw TransferError.uploadError("Error uploading the file: \(error)")
}
print("File uploaded to \(fileURL.path).")
}
```
------
# Use checksums to work with an Amazon S3 object using an AWS SDK
The following code example shows how to use checksums to work with an Amazon S3 object.
------
#### [ Java ]
**SDK for Java 2.x**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3#code-examples).
The code examples use a subset of the following imports.
```
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm;
import software.amazon.awssdk.services.s3.model.ChecksumMode;
import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload;
import software.amazon.awssdk.services.s3.model.CompletedPart;
import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.UploadPartRequest;
import software.amazon.awssdk.services.s3.model.UploadPartResponse;
import software.amazon.awssdk.services.s3.waiters.S3Waiter;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.FileUpload;
import software.amazon.awssdk.transfer.s3.model.UploadFileRequest;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.ByteBuffer;
import java.nio.file.Paths;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
import java.util.Objects;
import java.util.UUID;
```
Specify a checksum algorithm for the `putObject` method when you [build the `PutObjectRequest`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/PutObjectRequest.Builder.html).
```
public void putObjectWithChecksum() {
s3Client.putObject(b -> b
.bucket(bucketName)
.key(key)
.checksumAlgorithm(ChecksumAlgorithm.CRC32),
RequestBody.fromString("This is a test"));
}
```
Verify the checksum for the `getObject` method when you [build the GetObjectRequest](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/GetObjectRequest.Builder.html).
```
public GetObjectResponse getObjectWithChecksum() {
return s3Client.getObject(b -> b
.bucket(bucketName)
.key(key)
.checksumMode(ChecksumMode.ENABLED))
.response();
}
```
Pre-calculate a checksum for the `putObject` method when you [build the `PutObjectRequest`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/PutObjectRequest.Builder.html).
```
public void putObjectWithPrecalculatedChecksum(String filePath) {
String checksum = calculateChecksum(filePath, "SHA-256");
s3Client.putObject((b -> b
.bucket(bucketName)
.key(key)
.checksumSHA256(checksum)),
RequestBody.fromFile(Paths.get(filePath)));
}
```
Use the [S3 Transfer Manager](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html) on top of the [AWS CRT-based S3 client](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html) to transparently perform a multipart upload when the size of the content exceeds a threshold. The default threshold size is 8 MB.
You can specify a checksum algorithm for the SDK to use. By default, the SDK uses the CRC32 algorithm.
```
public void multipartUploadWithChecksumTm(String filePath) {
S3TransferManager transferManager = S3TransferManager.create();
UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
.putObjectRequest(b -> b
.bucket(bucketName)
.key(key)
.checksumAlgorithm(ChecksumAlgorithm.SHA1))
.source(Paths.get(filePath))
.build();
FileUpload fileUpload = transferManager.uploadFile(uploadFileRequest);
fileUpload.completionFuture().join();
transferManager.close();
}
```
Use the [S3Client API](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Client.html) or (S3AsyncClient API) to perform a multipart upload. If you specify an additional checksum, you must specify the algorithm to use on the initiation of the upload. You must also specify the algorithm for each part request and provide the checksum calculated for each part after it is uploaded.
```
public void multipartUploadWithChecksumS3Client(String filePath) {
ChecksumAlgorithm algorithm = ChecksumAlgorithm.CRC32;
// Initiate the multipart upload.
CreateMultipartUploadResponse createMultipartUploadResponse = s3Client.createMultipartUpload(b -> b
.bucket(bucketName)
.key(key)
.checksumAlgorithm(algorithm)); // Checksum specified on initiation.
String uploadId = createMultipartUploadResponse.uploadId();
// Upload the parts of the file.
int partNumber = 1;
List completedParts = new ArrayList<>();
ByteBuffer bb = ByteBuffer.allocate(1024 * 1024 * 5); // 5 MB byte buffer
try (RandomAccessFile file = new RandomAccessFile(filePath, "r")) {
long fileSize = file.length();
long position = 0;
while (position < fileSize) {
file.seek(position);
long read = file.getChannel().read(bb);
bb.flip(); // Swap position and limit before reading from the buffer.
UploadPartRequest uploadPartRequest = UploadPartRequest.builder()
.bucket(bucketName)
.key(key)
.uploadId(uploadId)
.checksumAlgorithm(algorithm) // Checksum specified on each part.
.partNumber(partNumber)
.build();
UploadPartResponse partResponse = s3Client.uploadPart(
uploadPartRequest,
RequestBody.fromByteBuffer(bb));
CompletedPart part = CompletedPart.builder()
.partNumber(partNumber)
.checksumCRC32(partResponse.checksumCRC32()) // Provide the calculated checksum.
.eTag(partResponse.eTag())
.build();
completedParts.add(part);
bb.clear();
position += read;
partNumber++;
}
} catch (IOException e) {
System.err.println(e.getMessage());
}
// Complete the multipart upload.
s3Client.completeMultipartUpload(b -> b
.bucket(bucketName)
.key(key)
.uploadId(uploadId)
.multipartUpload(CompletedMultipartUpload.builder().parts(completedParts).build()));
}
```
+ For API details, see the following topics in *AWS SDK for Java 2.x API Reference*.
+ [CompleteMultipartUpload](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/CompleteMultipartUpload)
+ [CreateMultipartUpload](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/CreateMultipartUpload)
+ [UploadPart](https://docs.aws.amazon.com/goto/SdkForJavaV2/s3-2006-03-01/UploadPart)
------
# Work with Amazon S3 object integrity features using an AWS SDK
The following code example shows how to work with S3 object integrity features.
------
#### [ C\$1\$1 ]
**SDK for C\$1\$1**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/cpp/example_code/s3/s3_object_integrity_workflow#code-examples).
Run an interactive scenario demonstrating Amazon S3 object integrity features.
```
//! Routine which runs the S3 object integrity workflow.
/*!
\param clientConfig: Aws client configuration.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::s3ObjectIntegrityWorkflow(
const Aws::S3::S3ClientConfiguration &clientConfiguration) {
/*
* Create a large file to be used for multipart uploads.
*/
if (!createLargeFileIfNotExists()) {
std::cerr << "Workflow exiting because large file creation failed." << std::endl;
return false;
}
Aws::String bucketName = TEST_BUCKET_PREFIX;
bucketName += Aws::Utils::UUID::RandomUUID();
bucketName = Aws::Utils::StringUtils::ToLower(bucketName.c_str());
bucketName.resize(std::min(bucketName.size(), MAX_BUCKET_NAME_LENGTH));
introductoryExplanations(bucketName);
if (!AwsDoc::S3::createBucket(bucketName, clientConfiguration)) {
std::cerr << "Workflow exiting because bucket creation failed." << std::endl;
return false;
}
Aws::S3::S3ClientConfiguration s3ClientConfiguration(clientConfiguration);
std::shared_ptr client = Aws::MakeShared("S3Client", s3ClientConfiguration);
printAsterisksLine();
std::cout << "Choose from one of the following checksum algorithms."
<< std::endl;
for (HASH_METHOD hashMethod = DEFAULT; hashMethod <= SHA256; ++hashMethod) {
std::cout << " " << hashMethod << " - " << stringForHashMethod(hashMethod)
<< std::endl;
}
HASH_METHOD chosenHashMethod = askQuestionForIntRange("Enter an index: ", DEFAULT,
SHA256);
gUseCalculatedChecksum = !askYesNoQuestion(
"Let the SDK calculate the checksum for you? (y/n) ");
printAsterisksLine();
std::cout << "The workflow will now upload a file using PutObject."
<< std::endl;
std::cout << "Object integrity will be verified using the "
<< stringForHashMethod(chosenHashMethod) << " algorithm."
<< std::endl;
if (gUseCalculatedChecksum) {
std::cout
<< "A checksum computed by this workflow will be used for object integrity verification,"
<< std::endl;
std::cout << "except for the TransferManager upload." << std::endl;
} else {
std::cout
<< "A checksum computed by the SDK will be used for object integrity verification."
<< std::endl;
}
pressEnterToContinue();
printAsterisksLine();
std::shared_ptr inputData =
Aws::MakeShared("SampleAllocationTag",
TEST_FILE,
std::ios_base::in |
std::ios_base::binary);
if (!*inputData) {
std::cerr << "Error unable to read file " << TEST_FILE << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
Hasher hasher;
HASH_METHOD putObjectHashMethod = chosenHashMethod;
if (putObjectHashMethod == DEFAULT) {
putObjectHashMethod = MD5; // MD5 is the default hash method for PutObject.
std::cout << "The default checksum algorithm for PutObject is "
<< stringForHashMethod(putObjectHashMethod)
<< std::endl;
}
// Demonstrate in code how the hash is computed.
if (!hasher.calculateObjectHash(*inputData, putObjectHashMethod)) {
std::cerr << "Error calculating hash for file " << TEST_FILE << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
Aws::String key = stringForHashMethod(putObjectHashMethod);
key += "_";
key += TEST_FILE_KEY;
Aws::String localHash = hasher.getBase64HashString();
// Upload the object with PutObject
if (!putObjectWithHash(bucketName, key, localHash, putObjectHashMethod,
inputData, chosenHashMethod == DEFAULT,
*client)) {
std::cerr << "Error putting file " << TEST_FILE << " to bucket "
<< bucketName << " with key " << key << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
Aws::String retrievedHash;
if (!retrieveObjectHash(bucketName, key,
putObjectHashMethod, retrievedHash,
nullptr, *client)) {
std::cerr << "Error getting file " << TEST_FILE << " from bucket "
<< bucketName << " with key " << key << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
explainPutObjectResults();
verifyHashingResults(retrievedHash, hasher,
"PutObject upload", putObjectHashMethod);
printAsterisksLine();
pressEnterToContinue();
key = "tr_";
key += stringForHashMethod(chosenHashMethod) + "_" + MULTI_PART_TEST_FILE;
introductoryTransferManagerUploadExplanations(key);
HASH_METHOD transferManagerHashMethod = chosenHashMethod;
if (transferManagerHashMethod == DEFAULT) {
transferManagerHashMethod = CRC32; // The default hash method for the TransferManager is CRC32.
std::cout << "The default checksum algorithm for TransferManager is "
<< stringForHashMethod(transferManagerHashMethod)
<< std::endl;
}
// Upload the large file using the transfer manager.
if (!doTransferManagerUpload(bucketName, key, transferManagerHashMethod, chosenHashMethod == DEFAULT,
client)) {
std::cerr << "Exiting because of an error in doTransferManagerUpload." << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
std::vector retrievedTransferManagerPartHashes;
Aws::String retrievedTransferManagerFinalHash;
// Retrieve all the hashes for the TransferManager upload.
if (!retrieveObjectHash(bucketName, key,
transferManagerHashMethod,
retrievedTransferManagerFinalHash,
&retrievedTransferManagerPartHashes, *client)) {
std::cerr << "Exiting because of an error in retrieveObjectHash for TransferManager." << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
AwsDoc::S3::Hasher locallyCalculatedFinalHash;
std::vector locallyCalculatedPartHashes;
// Calculate the hashes locally to demonstrate how TransferManager hashes are computed.
if (!calculatePartHashesForFile(transferManagerHashMethod, MULTI_PART_TEST_FILE,
UPLOAD_BUFFER_SIZE,
locallyCalculatedFinalHash,
locallyCalculatedPartHashes)) {
std::cerr << "Exiting because of an error in calculatePartHashesForFile." << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
verifyHashingResults(retrievedTransferManagerFinalHash,
locallyCalculatedFinalHash, "TransferManager upload",
transferManagerHashMethod,
retrievedTransferManagerPartHashes,
locallyCalculatedPartHashes);
printAsterisksLine();
key = "mp_";
key += stringForHashMethod(chosenHashMethod) + "_" + MULTI_PART_TEST_FILE;
multiPartUploadExplanations(key, chosenHashMethod);
pressEnterToContinue();
std::shared_ptr largeFileInputData =
Aws::MakeShared("SampleAllocationTag",
MULTI_PART_TEST_FILE,
std::ios_base::in |
std::ios_base::binary);
if (!largeFileInputData->good()) {
std::cerr << "Error unable to read file " << TEST_FILE << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
HASH_METHOD multipartUploadHashMethod = chosenHashMethod;
if (multipartUploadHashMethod == DEFAULT) {
multipartUploadHashMethod = MD5; // The default hash method for multipart uploads is MD5.
std::cout << "The default checksum algorithm for multipart upload is "
<< stringForHashMethod(putObjectHashMethod)
<< std::endl;
}
AwsDoc::S3::Hasher hashData;
std::vector partHashes;
if (!doMultipartUpload(bucketName, key,
multipartUploadHashMethod,
largeFileInputData, chosenHashMethod == DEFAULT,
hashData,
partHashes,
*client)) {
std::cerr << "Exiting because of an error in doMultipartUpload." << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
std::cout << "Finished multipart upload of with hash method " <<
stringForHashMethod(multipartUploadHashMethod) << std::endl;
std::cout << "Now we will retrieve the checksums from the server." << std::endl;
retrievedHash.clear();
std::vector retrievedPartHashes;
if (!retrieveObjectHash(bucketName, key,
multipartUploadHashMethod,
retrievedHash, &retrievedPartHashes, *client)) {
std::cerr << "Exiting because of an error in retrieveObjectHash for multipart." << std::endl;
cleanUp(bucketName, clientConfiguration);
return false;
}
verifyHashingResults(retrievedHash, hashData, "MultiPart upload",
multipartUploadHashMethod,
retrievedPartHashes, partHashes);
printAsterisksLine();
if (askYesNoQuestion("Would you like to delete the resources created in this workflow? (y/n)")) {
return cleanUp(bucketName, clientConfiguration);
} else {
std::cout << "The bucket " << bucketName << " was not deleted." << std::endl;
return true;
}
}
//! Routine which uploads an object to an S3 bucket with different object integrity hashing methods.
/*!
\param bucket: The name of the S3 bucket where the object will be uploaded.
\param key: The unique identifier (key) for the object within the S3 bucket.
\param hashData: The hash value that will be associated with the uploaded object.
\param hashMethod: The hashing algorithm to use when calculating the hash value.
\param body: The data content of the object being uploaded.
\param useDefaultHashMethod: A flag indicating whether to use the default hash method or the one specified in the hashMethod parameter.
\param client: The S3 client instance used to perform the upload operation.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::putObjectWithHash(const Aws::String &bucket, const Aws::String &key,
const Aws::String &hashData,
AwsDoc::S3::HASH_METHOD hashMethod,
const std::shared_ptr &body,
bool useDefaultHashMethod,
const Aws::S3::S3Client &client) {
Aws::S3::Model::PutObjectRequest request;
request.SetBucket(bucket);
request.SetKey(key);
if (!useDefaultHashMethod) {
if (hashMethod != MD5) {
request.SetChecksumAlgorithm(getChecksumAlgorithmForHashMethod(hashMethod));
}
}
if (gUseCalculatedChecksum) {
switch (hashMethod) {
case AwsDoc::S3::MD5:
request.SetContentMD5(hashData);
break;
case AwsDoc::S3::SHA1:
request.SetChecksumSHA1(hashData);
break;
case AwsDoc::S3::SHA256:
request.SetChecksumSHA256(hashData);
break;
case AwsDoc::S3::CRC32:
request.SetChecksumCRC32(hashData);
break;
case AwsDoc::S3::CRC32C:
request.SetChecksumCRC32C(hashData);
break;
default:
std::cerr << "Unknown hash method." << std::endl;
return false;
}
}
request.SetBody(body);
Aws::S3::Model::PutObjectOutcome outcome = client.PutObject(request);
body->seekg(0, body->beg);
if (outcome.IsSuccess()) {
std::cout << "Object successfully uploaded." << std::endl;
} else {
std::cerr << "Error uploading object." <<
outcome.GetError().GetMessage() << std::endl;
}
return outcome.IsSuccess();
}
// ! Routine which retrieves the hash value of an object stored in an S3 bucket.
/*!
\param bucket: The name of the S3 bucket where the object is stored.
\param key: The unique identifier (key) of the object within the S3 bucket.
\param hashMethod: The hashing algorithm used to calculate the hash value of the object.
\param[out] hashData: The retrieved hash.
\param[out] partHashes: The part hashes if available.
\param client: The S3 client instance used to retrieve the object.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::retrieveObjectHash(const Aws::String &bucket, const Aws::String &key,
AwsDoc::S3::HASH_METHOD hashMethod,
Aws::String &hashData,
std::vector *partHashes,
const Aws::S3::S3Client &client) {
Aws::S3::Model::GetObjectAttributesRequest request;
request.SetBucket(bucket);
request.SetKey(key);
if (hashMethod == MD5) {
Aws::Vector attributes;
attributes.push_back(Aws::S3::Model::ObjectAttributes::ETag);
request.SetObjectAttributes(attributes);
Aws::S3::Model::GetObjectAttributesOutcome outcome = client.GetObjectAttributes(
request);
if (outcome.IsSuccess()) {
const Aws::S3::Model::GetObjectAttributesResult &result = outcome.GetResult();
hashData = result.GetETag();
} else {
std::cerr << "Error retrieving object etag attributes." <<
outcome.GetError().GetMessage() << std::endl;
return false;
}
} else { // hashMethod != MD5
Aws::Vector attributes;
attributes.push_back(Aws::S3::Model::ObjectAttributes::Checksum);
request.SetObjectAttributes(attributes);
Aws::S3::Model::GetObjectAttributesOutcome outcome = client.GetObjectAttributes(
request);
if (outcome.IsSuccess()) {
const Aws::S3::Model::GetObjectAttributesResult &result = outcome.GetResult();
switch (hashMethod) {
case AwsDoc::S3::DEFAULT: // NOLINT(*-branch-clone)
break; // Default is not supported.
#pragma clang diagnostic push
#pragma ide diagnostic ignored "UnreachableCode"
case AwsDoc::S3::MD5:
break; // MD5 is not supported.
#pragma clang diagnostic pop
case AwsDoc::S3::SHA1:
hashData = result.GetChecksum().GetChecksumSHA1();
break;
case AwsDoc::S3::SHA256:
hashData = result.GetChecksum().GetChecksumSHA256();
break;
case AwsDoc::S3::CRC32:
hashData = result.GetChecksum().GetChecksumCRC32();
break;
case AwsDoc::S3::CRC32C:
hashData = result.GetChecksum().GetChecksumCRC32C();
break;
default:
std::cerr << "Unknown hash method." << std::endl;
return false;
}
} else {
std::cerr << "Error retrieving object checksum attributes." <<
outcome.GetError().GetMessage() << std::endl;
return false;
}
if (nullptr != partHashes) {
attributes.clear();
attributes.push_back(Aws::S3::Model::ObjectAttributes::ObjectParts);
request.SetObjectAttributes(attributes);
outcome = client.GetObjectAttributes(request);
if (outcome.IsSuccess()) {
const Aws::S3::Model::GetObjectAttributesResult &result = outcome.GetResult();
const Aws::Vector parts = result.GetObjectParts().GetParts();
for (const Aws::S3::Model::ObjectPart &part: parts) {
switch (hashMethod) {
case AwsDoc::S3::DEFAULT: // Default is not supported. NOLINT(*-branch-clone)
break;
case AwsDoc::S3::MD5: // MD5 is not supported.
break;
case AwsDoc::S3::SHA1:
partHashes->push_back(part.GetChecksumSHA1());
break;
case AwsDoc::S3::SHA256:
partHashes->push_back(part.GetChecksumSHA256());
break;
case AwsDoc::S3::CRC32:
partHashes->push_back(part.GetChecksumCRC32());
break;
case AwsDoc::S3::CRC32C:
partHashes->push_back(part.GetChecksumCRC32C());
break;
default:
std::cerr << "Unknown hash method." << std::endl;
return false;
}
}
} else {
std::cerr << "Error retrieving object attributes for object parts." <<
outcome.GetError().GetMessage() << std::endl;
return false;
}
}
}
return true;
}
//! Verifies the hashing results between the retrieved and local hashes.
/*!
\param retrievedHash The hash value retrieved from the remote source.
\param localHash The hash value calculated locally.
\param uploadtype The type of upload (e.g., "multipart", "single-part").
\param hashMethod The hashing method used (e.g., MD5, SHA-256).
\param retrievedPartHashes (Optional) The list of hashes for the individual parts retrieved from the remote source.
\param localPartHashes (Optional) The list of hashes for the individual parts calculated locally.
*/
void AwsDoc::S3::verifyHashingResults(const Aws::String &retrievedHash,
const Hasher &localHash,
const Aws::String &uploadtype,
HASH_METHOD hashMethod,
const std::vector &retrievedPartHashes,
const std::vector &localPartHashes) {
std::cout << "For " << uploadtype << " retrieved hash is " << retrievedHash << std::endl;
if (!retrievedPartHashes.empty()) {
std::cout << retrievedPartHashes.size() << " part hash(es) were also retrieved."
<< std::endl;
for (auto &retrievedPartHash: retrievedPartHashes) {
std::cout << " Part hash " << retrievedPartHash << std::endl;
}
}
Aws::String hashString;
if (hashMethod == MD5) {
hashString = localHash.getHexHashString();
if (!localPartHashes.empty()) {
hashString += "-" + std::to_string(localPartHashes.size());
}
} else {
hashString = localHash.getBase64HashString();
}
bool allMatch = true;
if (hashString != retrievedHash) {
std::cerr << "For " << uploadtype << ", the main hashes do not match" << std::endl;
std::cerr << "Local hash- '" << hashString << "'" << std::endl;
std::cerr << "Remote hash - '" << retrievedHash << "'" << std::endl;
allMatch = false;
}
if (hashMethod != MD5) {
if (localPartHashes.size() != retrievedPartHashes.size()) {
std::cerr << "For " << uploadtype << ", the number of part hashes do not match" << std::endl;
std::cerr << "Local number of hashes- '" << localPartHashes.size() << "'"
<< std::endl;
std::cerr << "Remote number of hashes - '"
<< retrievedPartHashes.size()
<< "'" << std::endl;
}
for (int i = 0; i < localPartHashes.size(); ++i) {
if (localPartHashes[i] != retrievedPartHashes[i]) {
std::cerr << "For " << uploadtype << ", the part hashes do not match for part " << i + 1
<< "." << std::endl;
std::cerr << "Local hash- '" << localPartHashes[i] << "'"
<< std::endl;
std::cerr << "Remote hash - '" << retrievedPartHashes[i] << "'"
<< std::endl;
allMatch = false;
}
}
}
if (allMatch) {
std::cout << "For " << uploadtype << ", locally and remotely calculated hashes all match!" << std::endl;
}
}
static void transferManagerErrorCallback(const Aws::Transfer::TransferManager *,
const std::shared_ptr &,
const Aws::Client::AWSError &err) {
std::cerr << "Error during transfer: '" << err.GetMessage() << "'" << std::endl;
}
static void transferManagerStatusCallback(const Aws::Transfer::TransferManager *,
const std::shared_ptr &handle) {
if (handle->GetStatus() == Aws::Transfer::TransferStatus::IN_PROGRESS) {
std::cout << "Bytes transferred: " << handle->GetBytesTransferred() << std::endl;
}
}
//! Routine which uploads an object to an S3 bucket using the AWS C++ SDK's Transfer Manager.
/*!
\param bucket: The name of the S3 bucket where the object will be uploaded.
\param key: The unique identifier (key) for the object within the S3 bucket.
\param hashMethod: The hashing algorithm to use when calculating the hash value.
\param useDefaultHashMethod: A flag indicating whether to use the default hash method or the one specified in the hashMethod parameter.
\param client: The S3 client instance used to perform the upload operation.
\return bool: Function succeeded.
*/
bool
AwsDoc::S3::doTransferManagerUpload(const Aws::String &bucket, const Aws::String &key,
AwsDoc::S3::HASH_METHOD hashMethod,
bool useDefaultHashMethod,
const std::shared_ptr &client) {
std::shared_ptr executor = Aws::MakeShared(
"executor", 25);
Aws::Transfer::TransferManagerConfiguration transfer_config(executor.get());
transfer_config.s3Client = client;
transfer_config.bufferSize = UPLOAD_BUFFER_SIZE;
if (!useDefaultHashMethod) {
if (hashMethod == MD5) {
transfer_config.computeContentMD5 = true;
} else {
transfer_config.checksumAlgorithm = getChecksumAlgorithmForHashMethod(
hashMethod);
}
}
transfer_config.errorCallback = transferManagerErrorCallback;
transfer_config.transferStatusUpdatedCallback = transferManagerStatusCallback;
std::shared_ptr transfer_manager = Aws::Transfer::TransferManager::Create(
transfer_config);
std::cout << "Uploading the file..." << std::endl;
std::shared_ptr uploadHandle = transfer_manager->UploadFile(MULTI_PART_TEST_FILE,
bucket, key,
"text/plain",
Aws::Map());
uploadHandle->WaitUntilFinished();
bool success =
uploadHandle->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED;
if (!success) {
Aws::Client::AWSError err = uploadHandle->GetLastError();
std::cerr << "File upload failed: " << err.GetMessage() << std::endl;
}
return success;
}
//! Routine which calculates the hash values for each part of a file being uploaded to an S3 bucket.
/*!
\param hashMethod: The hashing algorithm to use when calculating the hash values.
\param fileName: The path to the file for which the part hashes will be calculated.
\param bufferSize: The size of the buffer to use when reading the file.
\param[out] hashDataResult: The Hasher object that will store the concatenated hash value.
\param[out] partHashes: The vector that will store the calculated hash values for each part of the file.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::calculatePartHashesForFile(AwsDoc::S3::HASH_METHOD hashMethod,
const Aws::String &fileName,
size_t bufferSize,
AwsDoc::S3::Hasher &hashDataResult,
std::vector &partHashes) {
std::ifstream fileStream(fileName.c_str(), std::ifstream::binary);
fileStream.seekg(0, std::ifstream::end);
size_t objectSize = fileStream.tellg();
fileStream.seekg(0, std::ifstream::beg);
std::vector totalHashBuffer;
size_t uploadedBytes = 0;
while (uploadedBytes < objectSize) {
std::vector buffer(bufferSize);
std::streamsize bytesToRead = static_cast(std::min(buffer.size(), objectSize - uploadedBytes));
fileStream.read((char *) buffer.data(), bytesToRead);
Aws::Utils::Stream::PreallocatedStreamBuf preallocatedStreamBuf(buffer.data(),
bytesToRead);
std::shared_ptr body =
Aws::MakeShared("SampleAllocationTag",
&preallocatedStreamBuf);
Hasher hasher;
if (!hasher.calculateObjectHash(*body, hashMethod)) {
std::cerr << "Error calculating hash." << std::endl;
return false;
}
Aws::String base64HashString = hasher.getBase64HashString();
partHashes.push_back(base64HashString);
Aws::Utils::ByteBuffer hashBuffer = hasher.getByteBufferHash();
totalHashBuffer.insert(totalHashBuffer.end(), hashBuffer.GetUnderlyingData(),
hashBuffer.GetUnderlyingData() + hashBuffer.GetLength());
uploadedBytes += bytesToRead;
}
return hashDataResult.calculateObjectHash(totalHashBuffer, hashMethod);
}
//! Create a multipart upload.
/*!
\param bucket: The name of the S3 bucket where the object will be uploaded.
\param key: The unique identifier (key) for the object within the S3 bucket.
\param client: The S3 client instance used to perform the upload operation.
\return Aws::String: Upload ID or empty string if failed.
*/
Aws::String
AwsDoc::S3::createMultipartUpload(const Aws::String &bucket, const Aws::String &key,
Aws::S3::Model::ChecksumAlgorithm checksumAlgorithm,
const Aws::S3::S3Client &client) {
Aws::S3::Model::CreateMultipartUploadRequest request;
request.SetBucket(bucket);
request.SetKey(key);
if (checksumAlgorithm != Aws::S3::Model::ChecksumAlgorithm::NOT_SET) {
request.SetChecksumAlgorithm(checksumAlgorithm);
}
Aws::S3::Model::CreateMultipartUploadOutcome outcome =
client.CreateMultipartUpload(request);
Aws::String uploadID;
if (outcome.IsSuccess()) {
uploadID = outcome.GetResult().GetUploadId();
} else {
std::cerr << "Error creating multipart upload: " << outcome.GetError().GetMessage() << std::endl;
}
return uploadID;
}
//! Upload a part to an S3 bucket.
/*!
\param bucket: The name of the S3 bucket where the object will be uploaded.
\param key: The unique identifier (key) for the object within the S3 bucket.
\param uploadID: An upload ID string.
\param partNumber:
\param checksumAlgorithm: Checksum algorithm, ignored when NOT_SET.
\param calculatedHash: A data integrity hash to set, depending on the checksum algorithm,
ignored when it is an empty string.
\param body: An shared_ptr IOStream of the data to be uploaded.
\param client: The S3 client instance used to perform the upload operation.
\return UploadPartOutcome: The outcome.
*/
Aws::S3::Model::UploadPartOutcome AwsDoc::S3::uploadPart(const Aws::String &bucket,
const Aws::String &key,
const Aws::String &uploadID,
int partNumber,
Aws::S3::Model::ChecksumAlgorithm checksumAlgorithm,
const Aws::String &calculatedHash,
const std::shared_ptr &body,
const Aws::S3::S3Client &client) {
Aws::S3::Model::UploadPartRequest request;
request.SetBucket(bucket);
request.SetKey(key);
request.SetUploadId(uploadID);
request.SetPartNumber(partNumber);
if (checksumAlgorithm != Aws::S3::Model::ChecksumAlgorithm::NOT_SET) {
request.SetChecksumAlgorithm(checksumAlgorithm);
}
request.SetBody(body);
if (!calculatedHash.empty()) {
switch (checksumAlgorithm) {
case Aws::S3::Model::ChecksumAlgorithm::NOT_SET:
request.SetContentMD5(calculatedHash);
break;
case Aws::S3::Model::ChecksumAlgorithm::CRC32:
request.SetChecksumCRC32(calculatedHash);
break;
case Aws::S3::Model::ChecksumAlgorithm::CRC32C:
request.SetChecksumCRC32C(calculatedHash);
break;
case Aws::S3::Model::ChecksumAlgorithm::SHA1:
request.SetChecksumSHA1(calculatedHash);
break;
case Aws::S3::Model::ChecksumAlgorithm::SHA256:
request.SetChecksumSHA256(calculatedHash);
break;
}
}
return client.UploadPart(request);
}
//! Abort a multipart upload to an S3 bucket.
/*!
\param bucket: The name of the S3 bucket where the object will be uploaded.
\param key: The unique identifier (key) for the object within the S3 bucket.
\param uploadID: An upload ID string.
\param client: The S3 client instance used to perform the upload operation.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::abortMultipartUpload(const Aws::String &bucket,
const Aws::String &key,
const Aws::String &uploadID,
const Aws::S3::S3Client &client) {
Aws::S3::Model::AbortMultipartUploadRequest request;
request.SetBucket(bucket);
request.SetKey(key);
request.SetUploadId(uploadID);
Aws::S3::Model::AbortMultipartUploadOutcome outcome =
client.AbortMultipartUpload(request);
if (outcome.IsSuccess()) {
std::cout << "Multipart upload aborted." << std::endl;
} else {
std::cerr << "Error aborting multipart upload: " << outcome.GetError().GetMessage() << std::endl;
}
return outcome.IsSuccess();
}
//! Complete a multipart upload to an S3 bucket.
/*!
\param bucket: The name of the S3 bucket where the object will be uploaded.
\param key: The unique identifier (key) for the object within the S3 bucket.
\param uploadID: An upload ID string.
\param parts: A vector of CompleteParts.
\param client: The S3 client instance used to perform the upload operation.
\return CompleteMultipartUploadOutcome: The request outcome.
*/
Aws::S3::Model::CompleteMultipartUploadOutcome AwsDoc::S3::completeMultipartUpload(const Aws::String &bucket,
const Aws::String &key,
const Aws::String &uploadID,
const Aws::Vector &parts,
const Aws::S3::S3Client &client) {
Aws::S3::Model::CompletedMultipartUpload completedMultipartUpload;
completedMultipartUpload.SetParts(parts);
Aws::S3::Model::CompleteMultipartUploadRequest request;
request.SetBucket(bucket);
request.SetKey(key);
request.SetUploadId(uploadID);
request.SetMultipartUpload(completedMultipartUpload);
Aws::S3::Model::CompleteMultipartUploadOutcome outcome =
client.CompleteMultipartUpload(request);
if (!outcome.IsSuccess()) {
std::cerr << "Error completing multipart upload: " << outcome.GetError().GetMessage() << std::endl;
}
return outcome;
}
//! Routine which performs a multi-part upload.
/*!
\param bucket: The name of the S3 bucket where the object will be uploaded.
\param key: The unique identifier (key) for the object within the S3 bucket.
\param hashMethod: The hashing algorithm to use when calculating the hash value.
\param ioStream: An IOStream for the data to be uploaded.
\param useDefaultHashMethod: A flag indicating whether to use the default hash method or the one specified in the hashMethod parameter.
\param[out] hashDataResult: The Hasher object that will store the concatenated hash value.
\param[out] partHashes: The vector that will store the calculated hash values for each part of the file.
\param client: The S3 client instance used to perform the upload operation.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::doMultipartUpload(const Aws::String &bucket,
const Aws::String &key,
AwsDoc::S3::HASH_METHOD hashMethod,
const std::shared_ptr &ioStream,
bool useDefaultHashMethod,
AwsDoc::S3::Hasher &hashDataResult,
std::vector &partHashes,
const Aws::S3::S3Client &client) {
// Get object size.
ioStream->seekg(0, ioStream->end);
size_t objectSize = ioStream->tellg();
ioStream->seekg(0, ioStream->beg);
Aws::S3::Model::ChecksumAlgorithm checksumAlgorithm = Aws::S3::Model::ChecksumAlgorithm::NOT_SET;
if (!useDefaultHashMethod) {
if (hashMethod != MD5) {
checksumAlgorithm = getChecksumAlgorithmForHashMethod(hashMethod);
}
}
Aws::String uploadID = createMultipartUpload(bucket, key, checksumAlgorithm, client);
if (uploadID.empty()) {
return false;
}
std::vector totalHashBuffer;
bool uploadSucceeded = true;
std::streamsize uploadedBytes = 0;
int partNumber = 1;
Aws::Vector parts;
while (uploadedBytes < objectSize) {
std::cout << "Uploading part " << partNumber << "." << std::endl;
std::vector buffer(UPLOAD_BUFFER_SIZE);
std::streamsize bytesToRead = static_cast(std::min(buffer.size(),
objectSize - uploadedBytes));
ioStream->read((char *) buffer.data(), bytesToRead);
Aws::Utils::Stream::PreallocatedStreamBuf preallocatedStreamBuf(buffer.data(),
bytesToRead);
std::shared_ptr body =
Aws::MakeShared("SampleAllocationTag",
&preallocatedStreamBuf);
Hasher hasher;
if (!hasher.calculateObjectHash(*body, hashMethod)) {
std::cerr << "Error calculating hash." << std::endl;
uploadSucceeded = false;
break;
}
Aws::String base64HashString = hasher.getBase64HashString();
partHashes.push_back(base64HashString);
Aws::Utils::ByteBuffer hashBuffer = hasher.getByteBufferHash();
totalHashBuffer.insert(totalHashBuffer.end(), hashBuffer.GetUnderlyingData(),
hashBuffer.GetUnderlyingData() + hashBuffer.GetLength());
Aws::String calculatedHash;
if (gUseCalculatedChecksum) {
calculatedHash = base64HashString;
}
Aws::S3::Model::UploadPartOutcome uploadPartOutcome = uploadPart(bucket, key, uploadID, partNumber,
checksumAlgorithm, base64HashString, body,
client);
if (uploadPartOutcome.IsSuccess()) {
const Aws::S3::Model::UploadPartResult &uploadPartResult = uploadPartOutcome.GetResult();
Aws::S3::Model::CompletedPart completedPart;
completedPart.SetETag(uploadPartResult.GetETag());
completedPart.SetPartNumber(partNumber);
switch (hashMethod) {
case AwsDoc::S3::MD5:
break; // Do nothing.
case AwsDoc::S3::SHA1:
completedPart.SetChecksumSHA1(uploadPartResult.GetChecksumSHA1());
break;
case AwsDoc::S3::SHA256:
completedPart.SetChecksumSHA256(uploadPartResult.GetChecksumSHA256());
break;
case AwsDoc::S3::CRC32:
completedPart.SetChecksumCRC32(uploadPartResult.GetChecksumCRC32());
break;
case AwsDoc::S3::CRC32C:
completedPart.SetChecksumCRC32C(uploadPartResult.GetChecksumCRC32C());
break;
default:
std::cerr << "Unhandled hash method for completedPart." << std::endl;
break;
}
parts.push_back(completedPart);
} else {
std::cerr << "Error uploading part. " <<
uploadPartOutcome.GetError().GetMessage() << std::endl;
uploadSucceeded = false;
break;
}
uploadedBytes += bytesToRead;
partNumber++;
}
if (!uploadSucceeded) {
abortMultipartUpload(bucket, key, uploadID, client);
return false;
} else {
Aws::S3::Model::CompleteMultipartUploadOutcome completeMultipartUploadOutcome = completeMultipartUpload(bucket,
key,
uploadID,
parts,
client);
if (completeMultipartUploadOutcome.IsSuccess()) {
std::cout << "Multipart upload completed." << std::endl;
if (!hashDataResult.calculateObjectHash(totalHashBuffer, hashMethod)) {
std::cerr << "Error calculating hash." << std::endl;
return false;
}
} else {
std::cerr << "Error completing multipart upload." <<
completeMultipartUploadOutcome.GetError().GetMessage()
<< std::endl;
}
return completeMultipartUploadOutcome.IsSuccess();
}
}
//! Routine which retrieves the string for a HASH_METHOD constant.
/*!
\param: hashMethod: A HASH_METHOD constant.
\return: String: A string description of the hash method.
*/
Aws::String AwsDoc::S3::stringForHashMethod(AwsDoc::S3::HASH_METHOD hashMethod) {
switch (hashMethod) {
case AwsDoc::S3::DEFAULT:
return "Default";
case AwsDoc::S3::MD5:
return "MD5";
case AwsDoc::S3::SHA1:
return "SHA1";
case AwsDoc::S3::SHA256:
return "SHA256";
case AwsDoc::S3::CRC32:
return "CRC32";
case AwsDoc::S3::CRC32C:
return "CRC32C";
default:
return "Unknown";
}
}
//! Routine that returns the ChecksumAlgorithm for a HASH_METHOD constant.
/*!
\param: hashMethod: A HASH_METHOD constant.
\return: ChecksumAlgorithm: The ChecksumAlgorithm enum.
*/
Aws::S3::Model::ChecksumAlgorithm
AwsDoc::S3::getChecksumAlgorithmForHashMethod(AwsDoc::S3::HASH_METHOD hashMethod) {
Aws::S3::Model::ChecksumAlgorithm result = Aws::S3::Model::ChecksumAlgorithm::NOT_SET;
switch (hashMethod) {
case AwsDoc::S3::DEFAULT:
std::cerr << "getChecksumAlgorithmForHashMethod- DEFAULT is not valid." << std::endl;
break; // Default is not supported.
case AwsDoc::S3::MD5:
break; // Ignore MD5.
case AwsDoc::S3::SHA1:
result = Aws::S3::Model::ChecksumAlgorithm::SHA1;
break;
case AwsDoc::S3::SHA256:
result = Aws::S3::Model::ChecksumAlgorithm::SHA256;
break;
case AwsDoc::S3::CRC32:
result = Aws::S3::Model::ChecksumAlgorithm::CRC32;
break;
case AwsDoc::S3::CRC32C:
result = Aws::S3::Model::ChecksumAlgorithm::CRC32C;
break;
default:
std::cerr << "Unknown hash method." << std::endl;
break;
}
return result;
}
//! Routine which cleans up after the example is complete.
/*!
\param bucket: The name of the S3 bucket where the object was uploaded.
\param clientConfiguration: The client configuration for the S3 client.
\return bool: Function succeeded.
*/
bool AwsDoc::S3::cleanUp(const Aws::String &bucketName,
const Aws::S3::S3ClientConfiguration &clientConfiguration) {
Aws::Vector keysResult;
bool result = true;
if (AwsDoc::S3::listObjects(bucketName, keysResult, clientConfiguration)) {
if (!keysResult.empty()) {
result = AwsDoc::S3::deleteObjects(keysResult, bucketName,
clientConfiguration);
}
} else {
result = false;
}
return result && AwsDoc::S3::deleteBucket(bucketName, clientConfiguration);
}
//! Console interaction introducing the workflow.
/*!
\param bucketName: The name of the S3 bucket to use.
*/
void AwsDoc::S3::introductoryExplanations(const Aws::String &bucketName) {
std::cout
<< "Welcome to the Amazon Simple Storage Service (Amazon S3) object integrity workflow."
<< std::endl;
printAsterisksLine();
std::cout
<< "This workflow demonstrates how Amazon S3 uses checksum values to verify the integrity of data\n";
std::cout << "uploaded to Amazon S3 buckets" << std::endl;
std::cout
<< "The AWS SDK for C++ automatically handles checksums.\n";
std::cout
<< "By default it calculates a checksum that is uploaded with an object.\n"
<< "The default checksum algorithm for PutObject and MultiPart upload is an MD5 hash.\n"
<< "The default checksum algorithm for TransferManager uploads is a CRC32 checksum."
<< std::endl;
std::cout
<< "You can override the default behavior, requiring one of the following checksums,\n";
std::cout << "MD5, CRC32, CRC32C, SHA-1 or SHA-256." << std::endl;
std::cout << "You can also set the checksum hash value, instead of letting the SDK calculate the value."
<< std::endl;
std::cout
<< "For more information, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html."
<< std::endl;
std::cout
<< "This workflow will locally compute checksums for files uploaded to an Amazon S3 bucket,\n";
std::cout << "even when the SDK also computes the checksum." << std::endl;
std::cout
<< "This is done to provide demonstration code for how the checksums are calculated."
<< std::endl;
std::cout << "A bucket named '" << bucketName << "' will be created for the object uploads."
<< std::endl;
}
//! Console interaction which explains the PutObject results.
/*!
*/
void AwsDoc::S3::explainPutObjectResults() {
std::cout << "The upload was successful.\n";
std::cout << "If the checksums had not matched, the upload would have failed."
<< std::endl;
std::cout
<< "The checksums calculated by the server have been retrieved using the GetObjectAttributes."
<< std::endl;
std::cout
<< "The locally calculated checksums have been verified against the retrieved checksums."
<< std::endl;
}
//! Console interaction explaining transfer manager uploads.
/*!
\param objectKey: The key for the object being uploaded.
*/
void AwsDoc::S3::introductoryTransferManagerUploadExplanations(
const Aws::String &objectKey) {
std::cout
<< "Now the workflow will demonstrate object integrity for TransferManager multi-part uploads."
<< std::endl;
std::cout
<< "The AWS C++ SDK has a TransferManager class which simplifies multipart uploads."
<< std::endl;
std::cout
<< "The following code lets the TransferManager handle much of the checksum configuration."
<< std::endl;
std::cout << "An object with the key '" << objectKey
<< " will be uploaded by the TransferManager using a "
<< BUFFER_SIZE_IN_MEGABYTES << " MB buffer." << std::endl;
if (gUseCalculatedChecksum) {
std::cout << "For TransferManager uploads, this demo always lets the SDK calculate the hash value."
<< std::endl;
}
pressEnterToContinue();
printAsterisksLine();
}
//! Console interaction explaining multi-part uploads.
/*!
\param objectKey: The key for the object being uploaded.
\param chosenHashMethod: The hash method selected by the user.
*/
void AwsDoc::S3::multiPartUploadExplanations(const Aws::String &objectKey,
HASH_METHOD chosenHashMethod) {
std::cout
<< "Now we will provide an in-depth demonstration of multi-part uploading by calling the multi-part upload APIs directly."
<< std::endl;
std::cout << "These are the same APIs used by the TransferManager when uploading large files."
<< std::endl;
std::cout
<< "In the following code, the checksums are also calculated locally and then compared."
<< std::endl;
std::cout
<< "For multi-part uploads, a checksum is uploaded with each part. The final checksum is a concatenation of"
<< std::endl;
std::cout << "the checksums for each part." << std::endl;
std::cout
<< "This is explained in the user guide, https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html,\""
<< " in the section \"Using part-level checksums for multipart uploads\"." << std::endl;
std::cout << "Starting multipart upload of with hash method " <<
stringForHashMethod(chosenHashMethod) << " uploading to with object key\n"
<< "'" << objectKey << "'," << std::endl;
}
//! Create a large file for doing multi-part uploads.
/*!
*/
bool AwsDoc::S3::createLargeFileIfNotExists() {
// Generate a large file by writing this source file multiple times to a new file.
if (std::filesystem::exists(MULTI_PART_TEST_FILE)) {
return true;
}
std::ofstream newFile(MULTI_PART_TEST_FILE, std::ios::out
| std::ios::binary);
if (!newFile) {
std::cerr << "createLargeFileIfNotExists- Error creating file " << MULTI_PART_TEST_FILE <<
std::endl;
return false;
}
std::ifstream input(TEST_FILE, std::ios::in
| std::ios::binary);
if (!input) {
std::cerr << "Error opening file " << TEST_FILE <<
std::endl;
return false;
}
std::stringstream buffer;
buffer << input.rdbuf();
input.close();
while (newFile.tellp() < LARGE_FILE_SIZE && !newFile.bad()) {
buffer.seekg(std::stringstream::beg);
newFile << buffer.rdbuf();
}
newFile.close();
return true;
}
```
+ For API details, see the following topics in *AWS SDK for C\$1\$1 API Reference*.
+ [AbortMultipartUpload](https://docs.aws.amazon.com/goto/SdkForCpp/s3-2006-03-01/AbortMultipartUpload)
+ [CompleteMultipartUpload](https://docs.aws.amazon.com/goto/SdkForCpp/s3-2006-03-01/CompleteMultipartUpload)
+ [CreateMultipartUpload](https://docs.aws.amazon.com/goto/SdkForCpp/s3-2006-03-01/CreateMultipartUpload)
+ [DeleteObject](https://docs.aws.amazon.com/goto/SdkForCpp/s3-2006-03-01/DeleteObject)
+ [GetObjectAttributes](https://docs.aws.amazon.com/goto/SdkForCpp/s3-2006-03-01/GetObjectAttributes)
+ [PutObject](https://docs.aws.amazon.com/goto/SdkForCpp/s3-2006-03-01/PutObject)
+ [UploadPart](https://docs.aws.amazon.com/goto/SdkForCpp/s3-2006-03-01/UploadPart)
------
# Work with Amazon S3 versioned objects using an AWS SDK
The following code example shows how to:
+ Create a versioned S3 bucket.
+ Get all versions of an object.
+ Roll an object back to a previous version.
+ Delete and restore a versioned object.
+ Permanently delete all versions of an object.
------
#### [ Python ]
**SDK for Python (Boto3)**
There's more on GitHub. Find the complete example and learn how to set up and run in the [AWS Code Examples Repository](https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/python/example_code/s3/s3_versioning#code-examples).
Create functions that wrap S3 actions.
```
def create_versioned_bucket(bucket_name, prefix):
"""
Creates an Amazon S3 bucket, enables it for versioning, and configures a lifecycle
that expires noncurrent object versions after 7 days.
Adding a lifecycle configuration to a versioned bucket is a best practice.
It helps prevent objects in the bucket from accumulating a large number of
noncurrent versions, which can slow down request performance.
Usage is shown in the usage_demo_single_object function at the end of this module.
:param bucket_name: The name of the bucket to create.
:param prefix: Identifies which objects are automatically expired under the
configured lifecycle rules.
:return: The newly created bucket.
"""
try:
bucket = s3.create_bucket(
Bucket=bucket_name,
CreateBucketConfiguration={
"LocationConstraint": s3.meta.client.meta.region_name
},
)
logger.info("Created bucket %s.", bucket.name)
except ClientError as error:
if error.response["Error"]["Code"] == "BucketAlreadyOwnedByYou":
logger.warning("Bucket %s already exists! Using it.", bucket_name)
bucket = s3.Bucket(bucket_name)
else:
logger.exception("Couldn't create bucket %s.", bucket_name)
raise
try:
bucket.Versioning().enable()
logger.info("Enabled versioning on bucket %s.", bucket.name)
except ClientError:
logger.exception("Couldn't enable versioning on bucket %s.", bucket.name)
raise
try:
expiration = 7
bucket.LifecycleConfiguration().put(
LifecycleConfiguration={
"Rules": [
{
"Status": "Enabled",
"Prefix": prefix,
"NoncurrentVersionExpiration": {"NoncurrentDays": expiration},
}
]
}
)
logger.info(
"Configured lifecycle to expire noncurrent versions after %s days "
"on bucket %s.",
expiration,
bucket.name,
)
except ClientError as error:
logger.warning(
"Couldn't configure lifecycle on bucket %s because %s. "
"Continuing anyway.",
bucket.name,
error,
)
return bucket
def rollback_object(bucket, object_key, version_id):
"""
Rolls back an object to an earlier version by deleting all versions that
occurred after the specified rollback version.
Usage is shown in the usage_demo_single_object function at the end of this module.
:param bucket: The bucket that holds the object to roll back.
:param object_key: The object to roll back.
:param version_id: The version ID to roll back to.
"""
# Versions must be sorted by last_modified date because delete markers are
# at the end of the list even when they are interspersed in time.
versions = sorted(
bucket.object_versions.filter(Prefix=object_key),
key=attrgetter("last_modified"),
reverse=True,
)
logger.debug(
"Got versions:\n%s",
"\n".join(
[
f"\t{version.version_id}, last modified {version.last_modified}"
for version in versions
]
),
)
if version_id in [ver.version_id for ver in versions]:
print(f"Rolling back to version {version_id}")
for version in versions:
if version.version_id != version_id:
version.delete()
print(f"Deleted version {version.version_id}")
else:
break
print(f"Active version is now {bucket.Object(object_key).version_id}")
else:
raise KeyError(
f"{version_id} was not found in the list of versions for " f"{object_key}."
)
def revive_object(bucket, object_key):
"""
Revives a versioned object that was deleted by removing the object's active
delete marker.
A versioned object presents as deleted when its latest version is a delete marker.
By removing the delete marker, we make the previous version the latest version
and the object then presents as *not* deleted.
Usage is shown in the usage_demo_single_object function at the end of this module.
:param bucket: The bucket that contains the object.
:param object_key: The object to revive.
"""
# Get the latest version for the object.
response = s3.meta.client.list_object_versions(
Bucket=bucket.name, Prefix=object_key, MaxKeys=1
)
if "DeleteMarkers" in response:
latest_version = response["DeleteMarkers"][0]
if latest_version["IsLatest"]:
logger.info(
"Object %s was indeed deleted on %s. Let's revive it.",
object_key,
latest_version["LastModified"],
)
obj = bucket.Object(object_key)
obj.Version(latest_version["VersionId"]).delete()
logger.info(
"Revived %s, active version is now %s with body '%s'",
object_key,
obj.version_id,
obj.get()["Body"].read(),
)
else:
logger.warning(
"Delete marker is not the latest version for %s!", object_key
)
elif "Versions" in response:
logger.warning("Got an active version for %s, nothing to do.", object_key)
else:
logger.error("Couldn't get any version info for %s.", object_key)
def permanently_delete_object(bucket, object_key):
"""
Permanently deletes a versioned object by deleting all of its versions.
Usage is shown in the usage_demo_single_object function at the end of this module.
:param bucket: The bucket that contains the object.
:param object_key: The object to delete.
"""
try:
bucket.object_versions.filter(Prefix=object_key).delete()
logger.info("Permanently deleted all versions of object %s.", object_key)
except ClientError:
logger.exception("Couldn't delete all versions of %s.", object_key)
raise
```
Upload the stanza of a poem to a versioned object and perform a series of actions on it.
```
def usage_demo_single_object(obj_prefix="demo-versioning/"):
"""
Demonstrates usage of versioned object functions. This demo uploads a stanza
of a poem and performs a series of revisions, deletions, and revivals on it.
:param obj_prefix: The prefix to assign to objects created by this demo.
"""
with open("father_william.txt") as file:
stanzas = file.read().split("\n\n")
width = get_terminal_size((80, 20))[0]
print("-" * width)
print("Welcome to the usage demonstration of Amazon S3 versioning.")
print(
"This demonstration uploads a single stanza of a poem to an Amazon "
"S3 bucket and then applies various revisions to it."
)
print("-" * width)
print("Creating a version-enabled bucket for the demo...")
bucket = create_versioned_bucket("bucket-" + str(uuid.uuid1()), obj_prefix)
print("\nThe initial version of our stanza:")
print(stanzas[0])
# Add the first stanza and revise it a few times.
print("\nApplying some revisions to the stanza...")
obj_stanza_1 = bucket.Object(f"{obj_prefix}stanza-1")
obj_stanza_1.put(Body=bytes(stanzas[0], "utf-8"))
obj_stanza_1.put(Body=bytes(stanzas[0].upper(), "utf-8"))
obj_stanza_1.put(Body=bytes(stanzas[0].lower(), "utf-8"))
obj_stanza_1.put(Body=bytes(stanzas[0][::-1], "utf-8"))
print(
"The latest version of the stanza is now:",
obj_stanza_1.get()["Body"].read().decode("utf-8"),
sep="\n",
)
# Versions are returned in order, most recent first.
obj_stanza_1_versions = bucket.object_versions.filter(Prefix=obj_stanza_1.key)
print(
"The version data of the stanza revisions:",
*[
f" {version.version_id}, last modified {version.last_modified}"
for version in obj_stanza_1_versions
],
sep="\n",
)
# Rollback two versions.
print("\nRolling back two versions...")
rollback_object(bucket, obj_stanza_1.key, list(obj_stanza_1_versions)[2].version_id)
print(
"The latest version of the stanza:",
obj_stanza_1.get()["Body"].read().decode("utf-8"),
sep="\n",
)
# Delete the stanza
print("\nDeleting the stanza...")
obj_stanza_1.delete()
try:
obj_stanza_1.get()
except ClientError as error:
if error.response["Error"]["Code"] == "NoSuchKey":
print("The stanza is now deleted (as expected).")
else:
raise
# Revive the stanza
print("\nRestoring the stanza...")
revive_object(bucket, obj_stanza_1.key)
print(
"The stanza is restored! The latest version is again:",
obj_stanza_1.get()["Body"].read().decode("utf-8"),
sep="\n",
)
# Permanently delete all versions of the object. This cannot be undone!
print("\nPermanently deleting all versions of the stanza...")
permanently_delete_object(bucket, obj_stanza_1.key)
obj_stanza_1_versions = bucket.object_versions.filter(Prefix=obj_stanza_1.key)
if len(list(obj_stanza_1_versions)) == 0:
print("The stanza has been permanently deleted and now has no versions.")
else:
print("Something went wrong. The stanza still exists!")
print(f"\nRemoving {bucket.name}...")
bucket.delete()
print(f"{bucket.name} deleted.")
print("Demo done!")
```
+ For API details, see the following topics in *AWS SDK for Python (Boto3) API Reference*.
+ [CreateBucket](https://docs.aws.amazon.com/goto/boto3/s3-2006-03-01/CreateBucket)
+ [DeleteObject](https://docs.aws.amazon.com/goto/boto3/s3-2006-03-01/DeleteObject)
+ [ListObjectVersions](https://docs.aws.amazon.com/goto/boto3/s3-2006-03-01/ListObjectVersions)
+ [PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/goto/boto3/s3-2006-03-01/PutBucketLifecycleConfiguration)
------