Configuring transfers with other cloud object storage - AWS DataSync
Providing DataSync access to your other cloud object storageConsiderations when transferring from other cloud object storageCreating your DataSync agentCreating a transfer location for your other cloud object storageNext steps

Configuring transfers with other cloud object storage

With AWS DataSync, you can transfer data between AWS storage services and the following cloud object storage providers:

To set up this kind of transfer, you need to create a DataSync agent that can connect to your cloud object storage. You must also create a transfer location for your cloud object storage (specifically an Object storage location). DataSync can use this location as a source or destination for your transfer.

Providing DataSync access to your other cloud object storage

How DataSync accesses your cloud object storage depends on several factors, including whether your storage is compatible with the Amazon S3 API and the permissions and credentials that DataSync needs to access your storage.

Amazon S3 API compatibility

Your cloud object storage must be compatible with the following Amazon S3 API operations for DataSync to connect to it:

  • AbortMultipartUpload

  • CompleteMultipartUpload

  • CopyObject

  • CreateMultipartUpload

  • DeleteObject

  • DeleteObjects

  • DeleteObjectTagging

  • GetBucketLocation

  • GetObject

  • GetObjectTagging

  • HeadBucket

  • HeadObject

  • ListObjectsV2

  • PutObject

  • PutObjectTagging

  • UploadPart

Storage permissions and endpoints

You must configure the permissions that allow DataSync to access your cloud object storage. If your object storage is a source location, DataSync needs read and list permissions for the bucket that you're transferring data from. If your object storage is a destination location, DataSync needs read, list, write, and delete permissions for the bucket.

DataSync also needs an endpoint (or server) to connect to your storage. The following table describes the endpoints that DataSync can use to access other cloud object storage:

Other cloud provider Endpoint
Wasabi Cloud Storage

S3.region.wasabisys.com
DigitalOcean Spaces

region.digitaloceanspaces.com
Oracle Cloud Infrastructure Object Storage

namespace.compat.objectstorage.region.oraclecloud.com

Cloudflare R2 Storage

account-id.r2.cloudflarestorage.com

Backblaze B2 Cloud Storage

S3.region.backblazeb2.com
NAVER Cloud Object Storage

region.object.ncloudstorage.com (most regions)
Alibaba Cloud Object Storage Service

region.aliyuncs.com
IBM Cloud Object Storage

s3.region.cloud-object-storage.appdomain.cloud
Seagate Lyve Cloud

s3.region.lyvecloud.seagate.com
Important

For details on how to configure bucket permissions and updated information on storage endpoints, see your cloud provider's documentation.

Storage credentials

DataSync also needs the credentials to access the object storage bucket involved in your transfer. This might be an access key and secret key or something similar depending on how your cloud storage provider refers to these credentials.

For more information, see your cloud provider's documentation.

Considerations when transferring from other cloud object storage

When planning to transfer objects to or from another cloud storage provider by using DataSync, there are some things to keep in mind.

Costs

The fees associated with moving data in and out of another cloud storage provider can include:

  • Running an Amazon EC2 instance for your DataSync agent

  • Transferring the data by using DataSync, including request charges related to your cloud object storage and Amazon S3 (if S3 is your transfer destination)

  • Transferring data in or out of your cloud storage (check your cloud provider's pricing)

  • Storing data in an AWS storage service supported by DataSync

  • Storing data in another cloud provider (check your cloud provider's pricing)

Storage classes

Some cloud storage providers have storage classes (similar to Amazon S3) which DataSync can't read without being restored first. For example, Oracle Cloud Infrastructure Object Storage has an archive storage class. You need to restore objects in that storage class before DataSync can transfer them. For more information, see your cloud provider's documentation.

Object tags

Not all cloud providers support object tags. The ones that do might not allow querying tags through the Amazon S3 API. In either situation, your DataSync transfer task might fail if you try to copy object tags.

You can avoid this by clearing the Copy object tags checkbox in the DataSync console when creating, starting, or updating your task.

Transferring to Amazon S3

When transferring to Amazon S3, DataSync can't transfer objects larger than 5 TB. DataSync also can only copy object metadata up to 2 KB.

Creating your DataSync agent

To get started, you need a DataSync agent that can connect to your cloud object storage. This process includes deploying and activating an agent on an Amazon EC2 instance in your virtual private cloud (VPC) in AWS.

To create an Amazon EC2 agent

  1. Deploy an Amazon EC2 agent.

  2. Choose a service endpoint that the agent uses to communicate with AWS.

    In this situation, we recommend using a VPC service endpoint.

  3. Configure your network to work with VPC service endpoints.

  4. Activate the agent.

Creating a transfer location for your other cloud object storage

You can configure DataSync to use your cloud object storage as a source or destination location.

Before you begin

Make sure that you know how DataSync accesses your cloud object storage. You also need a DataSync agent that can connect to your cloud object storage.

  1. Open the AWS DataSync console at https://console.aws.amazon.com/datasync/.

  2. In the left navigation pane, expand Data transfer, then choose Locations and Create location.

  3. For Location type, choose Object storage.

  4. For Agents, choose the DataSync agent that can connect with your cloud object storage.

    You can choose more than one agent. For more information, see Using multiple DataSync agents.

  5. For Server, enter the endpoint that DataSync can use to access your cloud object storage:

    • Wasabi Cloud Storage – S3.region.wasabisys.com

    • DigitalOcean Spaces – region.digitaloceanspaces.com

    • Oracle Cloud Infrastructure Object Storage – namespace.compat.objectstorage.region.oraclecloud.com

    • Cloudflare R2 Storage – account-id.r2.cloudflarestorage.com

    • Backblaze B2 Cloud Storage – S3.region.backblazeb2.com

    • NAVER Cloud Object Storage – region.object.ncloudstorage.com (most regions)

    • Alibaba Cloud Object Storage Service – region.aliyuncs.com

    • IBM Cloud Object Storage – s3.region.cloud-object-storage.appdomain.cloud

    • Seagate Lyve Cloud – s3.region.lyvecloud.seagate.com

  6. For Bucket name, enter the name of the object storage bucket that you're transferring data to or from.

  7. Expand Additional settings. For Server protocol, choose HTTPS. For Server port, choose 443.

  8. Scroll down to the Authentication section. Make sure that the Requires credentials check box is selected, and then provide DataSync your storage credentials.

    • For Access key, enter the ID to access your cloud object storage.

    • For Secret key, enter the secret to access your cloud object storage.

  9. (Optional) Enter values for the Key and Value fields to tag the location.

    Tags help you manage, filter, and search for your AWS resources. We recommend creating at least a name tag for your location.

  10. Choose Create location.

Next steps

After you finish creating a DataSync location for your cloud object storage, you can continue setting up your transfer. Here are some next steps to consider:

  1. If you haven't already, create another location where you plan to transfer your data to or from in AWS.

  2. Learn how DataSync handles metadata and special files for object storage locations.

  3. Configure how your data gets transferred. For example, maybe you only want to transfer a subset of your data.

    Important

    Make sure that you configure how DataSync copies object tags correctly. For more information, see considerations with object tags.

  4. Start your transfer.