Pilih preferensi cookie Anda

Kami menggunakan cookie penting serta alat serupa yang diperlukan untuk menyediakan situs dan layanan. Kami menggunakan cookie performa untuk mengumpulkan statistik anonim sehingga kami dapat memahami cara pelanggan menggunakan situs dan melakukan perbaikan. Cookie penting tidak dapat dinonaktifkan, tetapi Anda dapat mengklik “Kustom” atau “Tolak” untuk menolak cookie performa.

Jika Anda setuju, AWS dan pihak ketiga yang disetujui juga akan menggunakan cookie untuk menyediakan fitur situs yang berguna, mengingat preferensi Anda, dan menampilkan konten yang relevan, termasuk iklan yang relevan. Untuk menerima atau menolak semua cookie yang tidak penting, klik “Terima” atau “Tolak”. Untuk membuat pilihan yang lebih detail, klik “Kustomisasi”.

Preferensi
Hubungi Kami
Umpan Balik
AWS Documentation
Buat Akun AWS
Architecture overview - Data Transfer from Amazon S3 Glacier Vaults to Amazon S3
Architecture diagram
Halaman ini belum diterjemahkan ke dalam bahasa Anda. Minta terjemahan

Architecture overview

PDF

This section provides a reference implementation architecture diagram for the components deployed with this solution.

Architecture diagram

Deploying this solution with the default parameters deploys the following components in your AWS account.

AWS Step Functions orchestrates Lambda functions that perform the data transfer between your S3 Glacier vault and an S3 bucket.

Data Transfer from Amazon S3 Glacier Vaults to Amazon S3 architecture on AWS

Note

AWS CloudFormation resources are created from AWS Cloud Development Kit (AWS CDK) (AWS CDK) constructs.

The high-level process flow for the solution components deployed with the AWS CloudFormation template is as follows:

  1. Customers invoke a transfer workflow by using a Systems Manager document (SSM document).

  2. The SSM document starts an AWS Step Functions Orchestrator execution.

  3. The Step Functions Orchestrator execution initiates a nested Step Functions Get Inventory workflow to retrieve the inventory file.

  4. Upon completion of the inventory retrieval, the solution invokes the Initiate Retrieval nested Step Functions workflow.

  5. When a job is ready, the Amazon S3 Glacier service sends a notification to an Amazon Simple Notification Service (Amazon SNS) topic indicating job completion.

  6. The solution stores all job completion notifications in the Amazon Simple Queue Service (Amazon SQS) Notifications queue.

  7. When an archive job is ready, the Amazon SQS Notifications queue invokes the AWS Lambda Notifications Processor function. This Lambda function prepares the initial steps for archive retrieval.

  8. The Lambda Notifications Processor function places chunks retrieval messages in Amazon SQS Chunks Retrieval queue for chunk processing.

  9. The Amazon SQS Chunks Retrieval queue invokes the Lambda Chunk Retrieval function to process each chunk.

  10. The Lambda Chunk Retrieval function downloads the chunk from the Amazon S3 Glacier service.

  11. The Lambda Chunk Retrieval function uploads a multipart upload part to the Amazon S3 service.

  12. After a new chunk is downloaded, the solution stores chunk metadata in Amazon DynamoDB (etag, checksum_sha_256, tree_checksum).

  13. The Lambda Chunk Retrieval function verifies whether all chunks for that archive have been processed. If yes, it inserts an event into the Amazon SQS Validation queue to invoke the Lambda Validate function.

  14. The Lambda Validate function does the following:

    1. Performs an integrity check against the tree hash in the inventory.

    2. Calculates a checksum and passes it to the into the close multipart upload call. If that hash is wrong, Amazon S3 rejects the request.

  15. A DynamoDB stream invokes the Lambda Metrics Processor function to update the transfer process metrics in DynamoDB.

  16. The Step Functions Orchestrator execution enters an async wait, pausing until the archive retrieval workflow concludes before initiating the Step Functions Cleanup workflow.

  17. The DynamoDB stream invokes the Lambda Async Facilitator function, which unlocks asynchronous waits in Step Functions.

  18. The Amazon EventBridge rules periodically initiate Step Functions Extend Download Window and Update CloudWatch Dashboard workflows.

  19. Customers monitor the transfer progress by using the Amazon CloudWatch dashboard.

Translation of S3 Glacier vault archive descriptions to S3 object names

To create the key name for each of the new objects in the Amazon S3 service, this solution uses the ArchiveDescription value for each ArchiveId listed in the Amazon S3 Glacier inventory file. The following are examples.

  1. If the ArchiveDescription is a single string value, such as data01, the solution translates that value to an S3 object key name in the destination S3 bucket.

    Amazon S3 object labeled data01.

  2. If the ArchiveDescription value is blank, then the solution does the following:

    1. Copies the archive.

    2. Uses the ArchiveId as the S3 object key name.

    3. Adds the prefix 00undefined to the S3 object key names and stores the objects in the destination S3 bucket.

      Amazon S3 object labeled with a randomized alpha-numeric name.

  3. If multiple ArchiveId entries have the same value for the ArchiveDescription field (for example, duplicatefile02.txt), then the solution appends a timestamp suffix to the name of the original file. This resolves the potential issue of having duplicate S3 object key names copied over one another. The timestamp used is the CreationDate of the archive.

Three Amazon S3 objects labeled duplicatefile02.txt with timestamps.

Creating custom file names for S3 objects

You can provide custom S3 object key names for each ArchiveId that's copied to your S3 bucket. To do this, provide a NamingOverrideFile to the solution when you launch the transfer workflow, using the NamingOverrideFile input parameter. Use the following process.

  1. Create a data file in CSV format. The file must contain only two columns: GlacierArchiveID and FileName (separated by a comma). The following table is an example.

    GlacierArchiveID FileName

    WVfrXME2KC6JIedfadJF937412-e

    Mydata.txt

    yLam5H76JXYSKKIY34404D-Kwcrk

    Myfolder/mydata2.txt

  2. Obtain a copy of your vault inventory file for the Amazon S3 Glacier service. For more information, see Downloading a Vault Inventory in Amazon S3 Glacier in the Amazon S3 Glacier Developer Guide.

  3. Copy all the ArchiveId values from your S3 Glacier vault inventory file. Paste them into the GlacierArchiveID column of your NamingOverride CSV file.

  4. In the FileName column, for each ArchiveID, enter your desired S3 object key name.

    Note

    If you provide an empty value for the FileName, the solution uses the original value for ArchiveDescription from the S3 Glacier archive.

  5. Upload the CSV file to any S3 bucket and create a presigned URL for the file.

  6. Use this presigned URL as the value of the NamingOverride File input parameter used when launching the transfer workflow.

Topik sebelumnya:

Concepts and definitions

Perlu bantuan?

PrivasiSyarat situsPreferensi cookie
© 2025, Amazon Web Services, Inc. atau afiliasinya. Semua hak dilindungi undang-undang.