Do I need an AWS DataSync agent? - AWS DataSync
Situations when you need a DataSync agentSituations when you don't need a DataSync agentUsing multiple DataSync agentsNext steps

Do I need an AWS DataSync agent?

To use AWS DataSync, you might need an agent. An agent is a virtual machine (VM) appliance that you deploy in your storage environment for data transfers or storage discovery.

Whether you need an agent depends on several factors, including the type of storage you're transferring to or from, if you're transferring across AWS accounts, and which AWS Regions you're transferring between. Before reading further, check that DataSync supports the transfer you're interested in.

Once you know that DataSync supports your transfer, review the following information to help you understand if you need an agent.

Situations when you need a DataSync agent

Most situations that require a DataSync agent involve storage that's managed by you or another cloud provider.

  • Transferring to or from on-premises storage

  • Transferring to or from other cloud storage

  • Transferring to or from edge storage

  • Transferring between some AWS storage services across AWS accounts (when neither storage service is Amazon S3)

    For more information, see Supported transfers across AWS accounts.

  • Transferring between a commercial AWS Region and AWS GovCloud (US) Region

  • Using AWS DataSync Discovery

Situations when you don't need a DataSync agent

The situations that don't require an agent apply whether you're transferring in the same AWS Region or across Regions.

  • Transferring between AWS storage services in the same AWS account

  • Transferring between an S3 bucket and a different AWS storage service across AWS accounts

Using multiple DataSync agents

You can use more than one DataSync agent with your data transfers. While most transfers only need one agent, using multiple agents can speed up transfers of large datasets with millions of files or objects.

In these situations, we recommend running transfer tasks in parallel. This approach spreads out the transfer workload across multiple tasks (each of which uses its own agent). It also helps reduce the time it takes DataSync to prepare and transfer your data. For more information, see Creating multiple tasks for transferring large datasets.

Another option—especially if you have millions of small files—is using multiple agents with a transfer location. For example, you can connect up to four agents to your on-premises Network File System (NFS) file service. This option can speed up your transfer, though the time it takes DataSync to prepare the transfer doesn’t change.

With either approach, be mindful that these can increase the I/O operations on your storage and affect your network bandwidth. For more information on using multiple agents for your DataSync transfers, see the AWS Storage Blog.

If you're thinking of using multiple agents, remember the following:

  • Using multiple agents with a location doesn't provide high availability. All the agents associated with a location must be online before you can start your transfer task. If one of the agents is offline, you can't run your task.

  • If you're using a virtual private cloud (VPC) service endpoint to communicate with the DataSync service, all the agents must use the same endpoint and subnet.

  • With DataSync Discovery, you can only use one agent per storage system.

Next steps

  • If you need an agent, review the agent requirements to understand what makes sense for your storage environment and what you need DataSync for.

  • If you don't need an agent for your transfer, you can start configuring your transfer.