Using S3CrtClient
for Amazon S3 operations
The S3CrtClient
class is available in version 1.9 of the AWS SDK for C++ and improves the throughput of uploading and downloading large data files to and from Amazon S3.
For more information on the improvements of this release, see Improving Amazon S3 Throughput with AWS SDK for C++ v1.9
The S3CrtClient
is implemented on the top of the AWS Common Runtime (CRT) libraries.
Note
To avoid being charged for incomplete or partial uploads, we recommend that you enable the AbortIncompleteMultipartUpload lifecycle rule on your Amazon S3 buckets.
This rule directs Amazon S3 to abort multipart uploads that don’t complete within a specified number of days after being initiated. When the set time limit is exceeded, Amazon S3 aborts the upload and then deletes the incomplete upload data.
For more information, see Setting lifecycle configuration on a bucket in the Amazon S3 User Guide.
Prerequisites
Before you begin, we recommend you read Getting started using the AWS SDK for C++.
Download the example code and build the solution as described in Get started on code examples.
To run the examples, the user profile your code uses to make the requests must have proper permissions in AWS (for the service and the action). For more information, see Providing AWS credentials.
Upload and download of object using S3CrtClient
This example demonstrates how to use the S3CrtClient
The provided data file, ny.json
, gets uploaded as a multipart upload in this example. This can be confirmed by viewing the debug logs
after a successful run of the program.
If the upload fails, an AbortMultipartUpload
is issued in the underlying CRT library to clean up any already-uploaded parts. However, not all
failures can be handled internally (such as a network cable being unplugged). It is recommended to create a lifecycle rule on your Amazon S3 bucket to ensure
partially uploaded data does not linger on your account (partially uploaded data is still billable). To find out how to set up a lifecycle rule, see Discovering and Deleting Incomplete
Multipart Uploads to Lower Amazon S3 Costs
Using the debug log to explore multipart upload details
-
In
main()
, note that there are "TODO" comments with instructions for updating the code.-
For
file_name
: From the link provided in the code comment download sample data fileny.json
, or use a large data file of your own. -
For
region
: Update theregion
variable, using the enum, to the AWS Region of your account. To find your Region of your account, log into the AWS Management Console, and locate the Region in the upper right-hand corner.
-
-
Build the example.
-
Copy the file specified by variable
file_name
to your executable folder and run thes3-crt-demo
executable. -
In your executable folder, find the most recent
.log
file. -
Open the log file, select search, and enter
partNumber
. -
The log contains entries similar to the following, where the
partNumber
anduploadId
are specified for each portion of the uploaded file:PUT /my-object partNumber=1&uploadId=gsk8vDbmnlA5EseDo._LDEgq22Qmt0SeuszYxMsZ9ABt503VqDIFOP8xzZI1P0zp.ToS.qo5kK16HNWogZF3KpRo.Dc7QnLZIK0BTmzCWwWoPax4T21hvP6nPdz9591F content-length:8388608 host:my-bucketasdfasdf.s3.us-east-2.amazonaws.com x-amz-content-sha256:UNSIGNED-PAYLOAD
and
PUT /my-object partNumber=2&uploadId=gsk8vDbmnlA5EseDo._LDEgq22Qmt0SeuszYxMsZ9ABt503VqDIFOP8xzZI1P0zp.ToS.qo5kK16HNWogZF3KpRo.Dc7QnLZIK0BTmzCWwWoPax4T21hvP6nPdz9591F content-length:8388608 host:my-bucketasdfasdf.s3.us-east-2.amazonaws.com x-amz-content-sha256:UNSIGNED-PAYLOAD
See the complete example