# Transferring to or from AWS storage with AWS DataSync
With AWS DataSync, you can transfer data to or from a number of AWS storage services. For more information, see [Where can I transfer my data with DataSync?](working-with-locations.md)
**Topics**
+ [Configuring AWS DataSync transfers with Amazon S3](create-s3-location.md)
+ [Configuring AWS DataSync transfers with Amazon EFS](create-efs-location.md)
+ [Configuring transfers with FSx for Windows File Server](create-fsx-location.md)
+ [Configuring DataSync transfers with FSx for Lustre](create-lustre-location.md)
+ [Configuring DataSync transfers with Amazon FSx for OpenZFS](create-openzfs-location.md)
+ [Configuring transfers with Amazon FSx for NetApp ONTAP](create-ontap-location.md)
# Configuring AWS DataSync transfers with Amazon S3
To transfer data to or from your Amazon S3 bucket, you create an AWS DataSync transfer *location*. DataSync can use this location as a source or destination for transferring data.
## Providing DataSync access to S3 buckets
DataSync needs access to the S3 bucket that you're transferring to or from. To do this, you must create an AWS Identity and Access Management (IAM) role that DataSync assumes with the permissions required to access the bucket. You then specify this role when [creating your Amazon S3 location for DataSync](#create-s3-location-how-to).
**Contents**
+ [Required permissions](#create-s3-location-required-permissions)
+ [Creating an IAM role for DataSync to access your Amazon S3 location](#create-role-manually)
+ [Accessing S3 buckets using server-side encryption](#create-s3-location-encryption)
+ [Accessing restricted S3 buckets](#denying-s3-access)
+ [Accessing S3 buckets with restricted VPC access](#create-s3-location-restricted-vpc)
### Required permissions
The permissions that your IAM role needs can depend on whether bucket is a DataSync source or destination location. Amazon S3 on Outposts requires a different set of permissions.
------
#### [ Amazon S3 (source location) ]
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::amzn-s3-demo-bucket"
},
{
"Action": [
"s3:GetObject",
"s3:GetObjectTagging",
"s3:GetObjectVersion",
"s3:GetObjectVersionTagging",
"s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*"
}
]
}
```
------
#### [ Amazon S3 (destination location) ]
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::amzn-s3-demo-bucket",
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "123456789012"
}
}
},
{
"Action": [
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:GetObject",
"s3:GetObjectTagging",
"s3:GetObjectVersion",
"s3:GetObjectVersionTagging",
"s3:ListMultipartUploadParts",
"s3:PutObject",
"s3:PutObjectTagging"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*",
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "123456789012"
}
}
}
]
}
```
------
#### [ Amazon S3 on Outposts ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"s3-outposts:ListBucket",
"s3-outposts:ListBucketMultipartUploads"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3-outposts:us-east-1:123456789012:outpost/outpost-id/bucket/amzn-s3-demo-bucket",
"arn:aws:s3-outposts:us-east-1:123456789012:outpost/outpost-id/accesspoint/bucket-access-point-name"
]
},
{
"Action": [
"s3-outposts:AbortMultipartUpload",
"s3-outposts:DeleteObject",
"s3-outposts:GetObject",
"s3-outposts:GetObjectTagging",
"s3-outposts:GetObjectVersion",
"s3-outposts:GetObjectVersionTagging",
"s3-outposts:ListMultipartUploadParts",
"s3-outposts:PutObject",
"s3-outposts:PutObjectTagging"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3-outposts:us-east-1:123456789012:outpost/outpost-id/bucket/amzn-s3-demo-bucket/*",
"arn:aws:s3-outposts:us-east-1:123456789012:outpost/outpost-id/accesspoint/bucket-access-point-name/*"
]
},
{
"Action": "s3-outposts:GetAccessPoint",
"Effect": "Allow",
"Resource": "arn:aws:s3-outposts:us-east-1:123456789012:outpost/outpost-id/accesspoint/bucket-access-point-name"
}
]
}
```
------
### Creating an IAM role for DataSync to access your Amazon S3 location
When [creating your Amazon S3 location](#create-s3-location-how-to) in the console, DataSync can automatically create and assume an IAM role that normally has the right permissions to access your S3 bucket.
In some situations, you might need to create this role manually (for example, accessing buckets with extra layers of security or transferring to or from a bucket in a different AWS accounts).
#### Manually creating an IAM role for DataSync
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the left navigation pane, under **Access management**, choose **Roles**, and then choose **Create role**.
1. On the **Select trusted entity** page, for **Trusted entity type**, choose **AWS service**.
1. For **Use case**, choose **DataSync** in the dropdown list and select **DataSync**. Choose **Next**.
1. On the **Add permissions** page, choose **Next**. Give your role a name and choose **Create role**.
1. On the **Roles** page, search for the role that you just created and choose its name.
1. On the role's details page, choose the **Permissions** tab. Choose **Add permissions** then **Create inline policy**.
1. Choose the **JSON** tab and [add the permissions required](#create-s3-location-required-permissions) to access your bucket into the policy editor.
1. Choose **Next**. Give your policy a name and choose **Create policy**.
1. (Recommended) To prevent the [cross-service confused deputy problem](cross-service-confused-deputy-prevention.md), do the following:
1. On the role's details page, choose the **Trust relationships** tab. Choose **Edit trust policy**.
1. Update the trust policy by using the following example, which includes the `aws:SourceArn` and `aws:SourceAccount` global condition context keys:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "datasync.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "444455556666"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:datasync:us-east-1:444455556666:*"
}
}
}
]
}
```
------
1. Choose **Update policy**.
You can specify this role when creating your Amazon S3 location.
### Accessing S3 buckets using server-side encryption
DataSync can transfer data to or from [S3 buckets that use server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html). The type of encryption key a bucket uses can determine if you need a custom policy allowing DataSync to access the bucket.
When using DataSync with S3 buckets that use server-side encryption, remember the following:
+ **If your S3 bucket is encrypted with an AWS managed key** – DataSync can access the bucket's objects by default if all your resources are in the same AWS account.
+ **If your S3 bucket is encrypted with a customer managed AWS Key Management Service (AWS KMS) key (SSE-KMS)** – The [key's policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html) must include the IAM role that DataSync uses to access the bucket.
+ **If your S3 bucket is encrypted with a customer managed SSE-KMS key and in a different AWS account** – DataSync needs permission to access the bucket in the other AWS account. You can set up this up by doing the following:
+ In the IAM role that DataSync uses, you must specify the cross-account bucket's SSE-KMS key by using the key's fully qualified Amazon Resource Name (ARN). This is the same key ARN that you use to configure the bucket's [default encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-bucket-encryption.html). You can't specify a key ID, alias name, or alias ARN in this situation.
Here's an example key ARN:
`arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab`
For more information on specifying KMS keys in IAM policy statements, see the *[AWS Key Management Service Developer Guide](https://docs.aws.amazon.com/kms/latest/developerguide/cmks-in-iam-policies.html)*.
+ In the SSE-KMS key policy, [specify the IAM role used by DataSync](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.html).
+ **If your S3 bucket is encrypted with a customer managed AWS KMS key (DSSE-KMS) for dual-layer server-side encryption** – The [key's policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html) must include the IAM role that DataSync uses to access the bucket. (Keep in mind that DSSE-KMS doesn't support [S3 Bucket Keys](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-key.html), which can reduce AWS KMS request costs.)
+ **If your S3 bucket is encrypted with a customer-provided encryption key (SSE-C)** – DataSync can't access this bucket.
#### Example: SSE-KMS key policy for DataSync
The following example is a [key policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) for a customer-managed SSE-KMS key. The policy is associated with an S3 bucket that uses server-side encryption.
If you want to use this example, replace the following values with your own:
+ *account-id* – Your AWS account.
+ *admin-role-name* – The name of the IAM role that can administer the key.
+ *datasync-role-name* – The name of the IAM role that allows DataSync to use the key when accessing the bucket.
------
#### [ JSON ]
****
```
{
"Id": "key-consolepolicy-3",
"Version":"2012-10-17",
"Statement": [
{
"Sid": "Enable IAM Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow access for Key Administrators",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/admin-role-name"
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:TagResource",
"kms:UntagResource",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
],
"Resource": "*"
},
{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/datasync-role-name"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*"
],
"Resource": "*"
}
]
}
```
------
### Accessing restricted S3 buckets
If you need to transfer to or from an S3 bucket that typically denies all access, you can edit the bucket policy so that DataSync can access the bucket only for your transfer.
#### Example: Allowing access based on IAM roles
1. Copy the following S3 bucket policy.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Sid": "Deny-access-to-bucket",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-bucket",
"arn:aws:s3:::amzn-s3-demo-bucket/*"
],
"Condition": {
"StringNotLike": {
"aws:userid": [
"datasync-iam-role-id:*",
"your-iam-role-id"
]
}
}
}]
}
```
------
1. In the policy, replace the following values:
+ `amzn-s3-demo-bucket` – Specify the name of the restricted S3 bucket.
+ `datasync-iam-role-id` – Specify the ID of the [IAM role that DataSync uses](#create-s3-location-access) to access the bucket.
Run the following AWS CLI command to get the IAM role ID:
`aws iam get-role --role-name datasync-iam-role-name`
In the output, look for the `RoleId` value:
`"RoleId": "ANPAJ2UCCR6DPCEXAMPLE"`
+ `your-iam-role-id` – Specify the ID of the IAM role that you use to create your DataSync location for the bucket.
Run the following command to get the IAM role ID:
`aws iam get-role --role-name your-iam-role-name`
In the output, look for the `RoleId` value:
`"RoleId": "AIDACKCEVSQ6C2EXAMPLE"`
1. [Add this policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) to your S3 bucket policy.
1. When you're done using DataSync with the restricted bucket, remove the conditions for both IAM roles from the bucket policy.
### Accessing S3 buckets with restricted VPC access
An Amazon S3 bucket that [limits access to specific virtual private cloud (VPC) endpoints or VPCs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies-vpc-endpoint.html) will deny DataSync from transferring to or from that bucket. To enable transfers in these situations, you can update the bucket's policy to include the IAM role that you [specify with your DataSync location](#create-s3-location-how-to).
------
#### [ Option 1: Allowing access based on DataSync location role ARN ]
In the S3 bucket policy, you can specify the Amazon Resource Name (ARN) of your DataSync location IAM role.
The following example is an S3 bucket policy that denies access from all but two VPCs (`vpc-1234567890abcdef0` and `vpc-abcdef01234567890`). However, the policy also includes the [ArnNotLikeIfExists](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html) condition and [aws:PrincipalArn](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-principalarn) condition key, which allow the ARN of a DataSync location role to access the bucket.
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "Access-to-specific-VPCs-only",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*",
"Condition": {
"StringNotEqualsIfExists": {
"aws:SourceVpc": [
"vpc-1234567890abcdef0",
"vpc-abcdef01234567890"
]
},
"ArnNotLikeIfExists": {
"aws:PrincipalArn": [
"arn:aws:iam::111122223333:role/datasync-location-role-name"
]
}
}
}
]
}
```
------
#### [ Option 2: Allowing access based on DataSync location role tag ]
In the S3 bucket policy, you can specify a tag attached to your DataSync location IAM role.
The following example is an S3 bucket policy that denies access from all but two VPCs (`vpc-1234567890abcdef0` and `vpc-abcdef01234567890`). However, the policy also includes the [StringNotEqualsIfExists](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html) condition and [aws:PrincipalTag](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-principaltag) condition key, which allow a principal with the tag key `exclude-from-vpc-restriction` and value `true`. You can try a similar approach in your bucket policy by specifying a tag attached to your DataSync location role.
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "Access-to-specific-VPCs-only",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::amzn-s3-demo-bucket/*",
"Condition": {
"StringNotEqualsIfExists": {
"aws:SourceVpc": [
"vpc-1234567890abcdef0",
"vpc-abcdef01234567890"
],
"aws:PrincipalTag/exclude-from-vpc-restriction": "true"
}
}
}
]
}
```
------
## Storage class considerations with Amazon S3 transfers
When Amazon S3 is your destination location, DataSync can transfer your data directly into a specific [Amazon S3 storage class](https://aws.amazon.com/s3/storage-classes/).
Some storage classes have behaviors that can affect your Amazon S3 storage costs. When using storage classes that can incur additional charges for overwriting, deleting, or retrieving objects, changes to object data or metadata result in such charges. For more information, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/).
**Important**
New objects transferred to your Amazon S3 destination location are stored using the storage class that you specify when [creating your location](#create-s3-location-how-to).
By default, DataSync preserves the storage class of existing objects in your destination location unless you configure your task to [transfer all data](configure-metadata.md#task-option-transfer-mode). In those situations, the storage class that you specify when creating your location is used for all objects.
| Amazon S3 storage class | Considerations |
| --- | --- |
| S3 Standard | Choose S3 Standard to store your frequently accessed files redundantly in multiple Availability Zones that are geographically separated. This is the default if you don't specify a storage class. |
| S3 Intelligent-Tiering | Choose S3 Intelligent-Tiering to optimize storage costs by automatically moving data to the most cost-effective storage access tier. You pay a monthly charge per object stored in the S3 Intelligent-Tiering storage class. This Amazon S3 charge includes monitoring data access patterns and moving objects between tiers. |
| S3 Standard-IA | Choose S3 Standard-IA to store your infrequently accessed objects redundantly in multiple Availability Zones that are geographically separated. Objects stored in the S3 Standard-IA storage class can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them. Changes to object data or metadata are equivalent to deleting an object and creating a new one to replace it. This results in additional charges for objects stored in the S3 Standard-IA storage class. Objects less than 128 KB are smaller than the minimum capacity charge per object in the S3 Standard-IA storage class. These objects are stored in the S3 Standard storage class. |
| S3 One Zone-IA | Choose S3 One Zone-IA to store your infrequently accessed objects in a single Availability Zone. Objects stored in the S3 One Zone-IA storage class can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them. Changes to object data or metadata are equivalent to deleting an object and creating a new one to replace it. This results in additional charges for objects stored in the S3 One Zone-IA storage class. Objects less than 128 KB are smaller than the minimum capacity charge per object in the S3 One Zone-IA storage class. These objects are stored in the S3 Standard storage class. |
| S3 Glacier Instant Retrieval | Choose S3 Glacier Instant Retrieval to archive objects that are rarely accessed but require retrieval in milliseconds. Data stored in the S3 Glacier Instant Retrieval storage class offers cost savings compared to the S3 Standard-IA storage class with the same latency and throughput performance. S3 Glacier Instant Retrieval has higher data access costs than S3 Standard-IA, though. Objects stored in S3 Glacier Instant Retrieval can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them. Changes to object data or metadata are equivalent to deleting an object and creating a new one to replace it. This results in additional charges for objects stored in the S3 Glacier Instant Retrieval storage class. Objects less than 128 KB are smaller than the minimum capacity charge per object in the S3 Glacier Instant Retrieval storage class. These objects are stored in the S3 Standard storage class. |
| S3 Glacier Flexible Retrieval | Choose S3 Glacier Flexible Retrieval for more active archives.Objects stored in S3 Glacier Flexible Retrieval can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them. Changes to object data or metadata are equivalent to deleting an object and creating a new one to replace it. This results in additional charges for objects stored in the S3 Glacier Flexible Retrieval storage class.The S3 Glacier Flexible Retrieval storage class requires 40 KB of additional metadata for each archived object. DataSync puts objects that are less than 40 KB in the S3 Standard storage class. You must restore objects archived in this storage class before DataSync can read them. For information, see [Working with archived objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/archived-objects.html) in the Amazon S3 User Guide.When using S3 Glacier Flexible Retrieval, choose the **Verify only the data transferred** task option to compare data and metadata checksums at the end of the transfer. You can't use the **Verify all data in the destination** option for this storage class because it requires retrieving all existing objects from the destination. |
| S3 Glacier Deep Archive | Choose S3 Glacier Deep Archive to archive your objects for long-term data retention and digital preservation where data is accessed once or twice a year. Objects stored in S3 Glacier Deep Archive can incur additional charges for overwriting, deleting, or retrieving. Consider how often these objects change, how long you plan to keep these objects, and how often you need to access them. Changes to object data or metadata are equivalent to deleting an object and creating a new one to replace it. This results in additional charges for objects stored in the S3 Glacier Deep Archive storage class. The S3 Glacier Deep Archive storage class requires 40 KB of additional metadata for each archived object. DataSync puts objects that are less than 40 KB in the S3 Standard storage class. You must restore objects archived in this storage class before DataSync can read them. For information, see [Working with archived objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/archived-objects.html) in the *Amazon S3 User Guide*. When using S3 Glacier Deep Archive, choose the **Verify only the data transferred** task option to compare data and metadata checksums at the end of the transfer. You can't use the **Verify all data in the destination** option for this storage class because it requires retrieving all existing objects from the destination. |
| S3 Outposts | The storage class for Amazon S3 on Outposts. |
## Evaluating S3 request costs when using DataSync
With Amazon S3 locations, you incur costs related to S3 API requests made by DataSync. This section can help you understand how DataSync uses these requests and how they might affect your [Amazon S3 costs](https://aws.amazon.com/s3/pricing/).
**Topics**
+ [S3 requests made by DataSync](#create-s3-location-s3-requests-made)
+ [Cost considerations](#create-s3-location-s3-requests-cost)
### S3 requests made by DataSync
The following table describes the S3 requests that DataSync can make when you’re copying data to or from an Amazon S3 location.
| S3 request | How DataSync uses it |
| --- | --- |
| [ListObjectV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) | DataSync makes at least one `LIST` request for every object ending in a forward slash (`/`) to list the objects that start with that prefix. This request is called during a task’s [preparing](run-task.md#understand-task-execution-statuses) phase. |
| [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) | DataSync makes `HEAD` requests to retrieve object metadata during a task’s [preparing](run-task.md#understand-task-execution-statuses) and [verifying](run-task.md#understand-task-execution-statuses) phases. There can be multiple `HEAD` requests per object depending on how you want DataSync to [verify the integrity of the data it transfers](configure-data-verification-options.md). |
| [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) | DataSync makes `GET` requests to read data from an object during a task’s [transferring](run-task.md#understand-task-execution-statuses) phase. There can be multiple `GET` requests for large objects. |
| [GetObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html) | If you configure your task to [copy object tags](configure-metadata.md), DataSync makes these `GET` requests to check for object tags during the task's [preparing](run-task.md#understand-task-execution-statuses) and [transferring](run-task.md#understand-task-execution-statuses) phases. |
| [PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) | DataSync makes `PUT` requests to create objects and prefixes in a destination S3 bucket during a task’s [transferring](run-task.md#understand-task-execution-statuses) phase. Since DataSync uses the [Amazon S3 multipart upload feature](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html), there can be multiple `PUT` requests for large objects. To help minimize storage costs, we recommend using a [lifecycle configuration]() to stop incomplete multipart uploads. |
| [PutObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html) | If your source objects have tags and you configure your task to [copy object tags](configure-metadata.md), DataSync makes these `PUT` requests when [transferring](run-task.md#understand-task-execution-statuses) those tags. |
| [CopyObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) | DataSync makes a `COPY` request to create a copy of an object only if that object’s metadata changes. This can happen if you originally copied data to the S3 bucket using another service or tool that didn’t carry over its metadata. |
### Cost considerations
DataSync makes S3 requests on S3 buckets every time you run your task. This can lead to charges adding up in certain situations. For example:
+ You’re frequently transferring objects to or from an S3 bucket.
+ You may not be transferring much data, but your S3 bucket has lots of objects in it. You can still see high charges in this scenario because DataSync makes S3 requests on each of the bucket's objects.
+ You're transferring between S3 buckets, so DataSync is making S3 requests on the source and destination.
To help minimize S3 request costs related to DataSync, consider the following:
**Topics**
+ [What S3 storage classes am I using?](#create-s3-location-s3-requests-storage-classes)
+ [How often do I need to transfer my data?](#create-s3-location-s3-requests-recurring-transfers)
#### What S3 storage classes am I using?
S3 request charges can vary based on the Amazon S3 storage class your objects are using, particularly for classes that archive objects (such as S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive).
Here are some scenarios in which storage classes can affect your S3 request charges when using DataSync:
+ Each time you run a task, DataSync makes `HEAD` requests to retrieve object metadata. These requests result in charges even if you aren’t moving any objects. How much these requests affect your bill depends on the storage class your objects are using along with the number of objects that DataSync scans.
+ If you moved objects into the S3 Glacier Instant Retrieval storage class (either directly or through a bucket lifecycle configuration), requests on objects in this class are more expensive than objects in other storage classes.
+ If you configure your DataSync task to [verify that your source and destination locations are fully synchronized](configure-data-verification-options.md), there will be `GET` requests for each object in all storage classes (except S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive).
+ In addition to `GET` requests, you incur data retrieval costs for objects in the S3 Standard-IA, S3 One Zone-IA, or S3 Glacier Instant Retrieval storage class.
For more information, see [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/).
#### How often do I need to transfer my data?
If you need to move data on a recurring basis, think about a [schedule](task-scheduling.md) that doesn't run more tasks than you need.
You may also consider limiting the scope of your transfers. For example, you can configure DataSync to focus on objects in certain prefixes or [filter what data gets transferred](filtering.md). These options can help reduce the number of S3 requests made each time you run your DataSync task.
## Object considerations with Amazon S3 transfers
+ If you're transferring from an S3 bucket, use [S3 Storage Lens](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage_lens_basics_metrics_recommendations.html) to determine how many objects you're moving.
+ When transferring between S3 buckets, we recommend using [Enhanced task mode](choosing-task-mode.md) because you aren't subject to DataSync task [quotas](datasync-limits.md).
+ DataSync might not transfer an object with nonstandard characters in its name. For more information, see the [object key naming guidelines](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html#object-key-guidelines) in the *Amazon S3 User Guide*.
+ When using DataSync with an S3 bucket that uses [versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html), remember the following:
+ When transferring to an S3 bucket, DataSync creates a new version of an object if that object is modified at the source. This results in additional charges.
+ An object has different version IDs in the source and destination buckets.
+ Only the most recent version of each object is transferred from the source bucket. Earlier versions are not copied to the destination.
+ After initially transferring data from an S3 bucket to a file system (for example, NFS or Amazon FSx), subsequent runs of the same DataSync task won't include objects that have been modified but are the same size they were during the first transfer.
## Creating your transfer location for an Amazon S3 general purpose bucket
To create a location for your transfer, you need an existing S3 general purpose bucket. If you don't have one, see the [https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html).
**Important**
Before you create your location, make sure that you read the following sections:
[Storage class considerations with Amazon S3 transfers](#using-storage-classes)
[Evaluating S3 request costs when using DataSync](#create-s3-location-s3-requests)
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Locations** and **Create location**.
1. For **Location type**, choose **Amazon S3**, and then choose **General purpose bucket**.
1. For **S3 URI**, enter or choose the bucket and prefix that you want to use for your location.
**Warning**
DataSync can't transfer objects with a prefix that begins with a slash (`/`) or includes `//`, `/./`, or `/../` patterns. For example:
`/photos`
`photos//2006/January`
`photos/./2006/February`
`photos/../2006/March`
1. For **S3 storage class when used as a destination**, choose a storage class that you want your objects to use when Amazon S3 is a transfer destination.
For more information, see [Storage class considerations with Amazon S3 transfers](#using-storage-classes).
1. For **IAM role**, do one of the following:
+ Choose **Autogenerate** for DataSync to automatically create an IAM role with the permissions required to access the S3 bucket.
If DataSync previously created an IAM role for this S3 bucket, that role is chosen by default.
+ Choose a custom IAM role that you created. For more information, see [Creating an IAM role for DataSync to access your Amazon S3 location](#create-role-manually).
1. (Optional) Choose **Add new tag** to tag your Amazon S3 location.
Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.
1. Choose **Create location**.
### Using the AWS CLI
1. Copy the following `create-location-s3` command:
```
aws datasync create-location-s3 \
--s3-bucket-arn 'arn:aws:s3:::amzn-s3-demo-bucket' \
--s3-storage-class 'your-S3-storage-class' \
--s3-config 'BucketAccessRoleArn=arn:aws:iam::account-id:role/role-allowing-datasync-operations' \
--subdirectory /your-prefix-name
```
1. For `--s3-bucket-arn`, specify the ARN of the S3 bucket that you want to use as a location.
1. For `--s3-storage-class`, specify a storage class that you want your objects to use when Amazon S3 is a transfer destination.
1. For `--s3-config`, specify the ARN of the IAM role that DataSync needs to access your bucket.
For more information, see [Creating an IAM role for DataSync to access your Amazon S3 location](#create-role-manually).
1. For `--subdirectory`, specify a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).
**Warning**
DataSync can't transfer objects with a prefix that begins with a slash (`/`) or includes `//`, `/./`, or `/../` patterns. For example:
`/photos`
`photos//2006/January`
`photos/./2006/February`
`photos/../2006/March`
1. Run the `create-location-s3` command.
If the command is successful, you get a response that shows you the ARN of the location that you created. For example:
```
{
"LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d"
}
```
You can use this location as a source or destination for your DataSync task.
## Creating your transfer location for an S3 on Outposts bucket
To create a location for your transfer, you need an existing Amazon S3 on Outposts bucket. If you don't have one, see the [https://docs.aws.amazon.com/AmazonS3/latest/s3-outposts/S3onOutposts.html](https://docs.aws.amazon.com/AmazonS3/latest/s3-outposts/S3onOutposts.html).
You also need a DataSync agent. For more information, see [Deploying your Basic mode agent on AWS Outposts](deploy-agents.md#outposts-agent).
When transferring from an S3 on Outposts bucket prefix that contains a large dataset (such as hundreds of thousands or millions of objects), your DataSync task might time out. To avoid this, consider using a [DataSync manifest](transferring-with-manifest.md), which lets you specify the exact objects that you need to transfer.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Locations** and **Create location**.
1. For **Location type**, choose **Amazon S3**, and then choose **Outposts bucket**.
1. For **S3 bucket**, choose an Amazon S3 access point that can access your S3 on Outposts bucket.
For more information, see the [https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points.html).
1. For **S3 storage class when used as a destination**, choose a storage class that you want your objects to use when Amazon S3 is a transfer destination.
For more information, see [Storage class considerations with Amazon S3 transfers](#using-storage-classes). DataSync by default uses the S3 Outposts storage class for Amazon S3 on Outposts.
1. For **Agents**, specify the Amazon Resource Name (ARN) of the DataSync agent on your Outpost.
1. For **Folder**, enter a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).
**Warning**
DataSync can't transfer objects with a prefix that begins with a slash (`/`) or includes `//`, `/./`, or `/../` patterns. For example:
`/photos`
`photos//2006/January`
`photos/./2006/February`
`photos/../2006/March`
1. For **IAM role**, do one of the following:
+ Choose **Autogenerate** for DataSync to automatically create an IAM role with the permissions required to access the S3 bucket.
If DataSync previously created an IAM role for this S3 bucket, that role is chosen by default.
+ Choose a custom IAM role that you created. For more information, see [Creating an IAM role for DataSync to access your Amazon S3 location](#create-role-manually).
1. (Optional) Choose **Add new tag** to tag your Amazon S3 location.
Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.
1. Choose **Create location**.
### Using the AWS CLI
1. Copy the following `create-location-s3` command:
```
aws datasync create-location-s3 \
--s3-bucket-arn 'bucket-access-point' \
--s3-storage-class 'your-S3-storage-class' \
--s3-config 'BucketAccessRoleArn=arn:aws:iam::account-id:role/role-allowing-datasync-operations' \
--subdirectory /your-folder \
--agent-arns 'arn:aws:datasync:your-region:account-id::agent/agent-agent-id'
```
1. For `--s3-bucket-arn`, specify the ARN an Amazon S3 access point that can access your S3 on Outposts bucket.
For more information, see the [https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points.html).
1. For `--s3-storage-class`, specify a storage class that you want your objects to use when Amazon S3 is a transfer destination.
For more information, see [Storage class considerations with Amazon S3 transfers](#using-storage-classes). DataSync by default uses the S3 Outposts storage class for S3 on Outposts.
1. For `--s3-config`, specify the ARN of the IAM role that DataSync needs to access your bucket.
For more information, see [Creating an IAM role for DataSync to access your Amazon S3 location](#create-role-manually).
1. For `--subdirectory`, specify a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).
**Warning**
DataSync can't transfer objects with a prefix that begins with a slash (`/`) or includes `//`, `/./`, or `/../` patterns. For example:
`/photos`
`photos//2006/January`
`photos/./2006/February`
`photos/../2006/March`
1. For `--agent-arns`, specify the ARN of the DataSync agent on your Outpost.
1. Run the `create-location-s3` command.
If the command is successful, you get a response that shows you the ARN of the location that you created. For example:
```
{
"LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d"
}
```
You can use this location as a source or destination for your DataSync task.
## Amazon S3 transfers across AWS accounts
With DataSync, you can move data to or from S3 buckets in [different AWS accounts](working-with-locations.md#working-with-locations-across-accounts). For more information, see the following tutorials:
+ [Transferring data from on-premises storage to Amazon S3 across AWS accounts](s3-cross-account-transfer.md)
+ [Transferring data from Amazon S3 to Amazon S3 across AWS accounts](tutorial_s3-s3-cross-account-transfer.md)
## Amazon S3 transfers between commercial and AWS GovCloud (US) Regions
By default, DataSync doesn't transfer between S3 buckets in commercial and AWS GovCloud (US) Regions. You can still set up this kind of transfer, though, by creating an object storage location for one of the S3 buckets in your transfer. You can perform this type of transfer with or without an agent. If you use an agent, your task must be configured for **Basic** mode. To transfer without an agent, you must use **Enhanced** mode.
**Before you begin**: Make sure that you understand the cost implications of transferring between Regions. For more information, see [AWS DataSync Pricing](https://aws.amazon.com/datasync/pricing/).
**Contents**
+ [Providing DataSync access to your object storage location's bucket](#create-s3-location-govcloud-iam)
+ [Creating your DataSync agent (optional)](#create-s3-location-govcloud-create-agent)
+ [Creating an object storage location for your S3 bucket](#create-s3-location-govcloud-how-to)
### Providing DataSync access to your object storage location's bucket
When creating the object storage location for this transfer, you must provide DataSync the credentials of an IAM user with permission to access the location's S3 bucket. For more information, see [Required permissions](#create-s3-location-required-permissions).
**Warning**
IAM users have long-term credentials, which presents a security risk. To help mitigate this risk, we recommend that you provide these users with only the permissions they require to perform the task and that you remove these users when they are no longer needed.
### Creating your DataSync agent (optional)
If you want to run your transfer using **Basic** mode, then you will need to use an agent. Because you're transferring between a commercial and AWS GovCloud (US) Region, you deploy your DataSync agent as an Amazon EC2 instance in one of the Regions. We recommend that your agent use a VPC service endpoint to avoid data transfer charges out to the public internet. For more information, see [Amazon EC2 Data Transfer pricing](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer).
Choose one of the following scenarios that describe how to create an agent based on the Region where you plan to run your DataSync task.
#### When running a DataSync task in a commercial Region
The following diagram shows a transfer where your DataSync task and agent are in the commercial Region.
![\[A DataSync agent deployed in a commercial Region for a cross-Region transfer to an S3 bucket in an AWS GovCloud (US) Region.\]](http://docs.aws.amazon.com/datasync/latest/userguide/images/s3-task-in-commercial.png)
| Reference | Description |
| --- | --- |
| 1 | In the commercial Region where you're running a DataSync task, data transfers from the source S3 bucket. The source bucket is configured as an [Amazon S3 location](#create-s3-location-how-to) in the commercial Region. |
| 2 | Data transfers through the DataSync agent, which is in the same VPC and subnet where the VPC service endpoint and [network interfaces](required-network-interfaces.md) are located. |
| 3 | Data transfers to the destination S3 bucket in the AWS GovCloud (US) Region. The destination bucket is configured as an [object storage location](#create-s3-location-govcloud-how-to) in the commercial Region. |
You can use this same setup to transfer the opposite direction, too, from the AWS GovCloud (US) Region to the commercial Region.
**To create your DataSync agent**
1. [Deploy an Amazon EC2 agent](deploy-agents.md#ec2-deploy-agent-how-to) in your commercial Region.
1. Configure your agent to use a [VPC service endpoint](choose-service-endpoint.md#datasync-in-vpc).
1. [Activate your agent](activate-agent.md).
#### When running a DataSync task in a GovCloud (US) Region
The following diagram shows a transfer where your DataSync task and agent are in the AWS GovCloud (US) Region.
![\[A DataSync agent deployed in a AWS GovCloud (US) Region or a cross-Region transfer to an S3 bucket in the same AWS GovCloud (US) Region.\]](http://docs.aws.amazon.com/datasync/latest/userguide/images/s3-task-in-govcloud-1.png)
| Reference | Description |
| --- | --- |
| 1 | Data transfers from the source S3 bucket in the commercial Region to the AWS GovCloud (US) Region where you're running a DataSync task. The source bucket is configured as an [object storage location](#create-s3-location-govcloud-how-to) in the AWS GovCloud (US) Region. |
| 2 | In the AWS GovCloud (US) Region, data transfers through the DataSync agent in the same VPC and subnet where the VPC service endpoint and [network interfaces](required-network-interfaces.md) are located. |
| 3 | Data transfers to the destination S3 bucket in the AWS GovCloud (US) Region. The destination bucket is configured as an [Amazon S3 location](#create-s3-location-how-to) in the AWS GovCloud (US) Region. |
You can use this same setup to transfer the opposite direction, too, from the AWS GovCloud (US) Region to the commercial Region.
**To create your DataSync agent**
1. [Deploy an Amazon EC2 agent](deploy-agents.md#ec2-deploy-agent-how-to) in your AWS GovCloud (US) Region.
1. Configure your agent to use a [VPC service endpoint](choose-service-endpoint.md#datasync-in-vpc).
1. [Activate your agent](activate-agent.md).
If your dataset is highly compressible, you might see reduced costs by instead creating your agent in a commercial Region while running a task in an AWS GovCloud (US) Region. There's more setup than normal for creating this agent, including preparing the agent for use in a commercial Region. For information about creating an agent for this setup, see the [Move data in and out of AWS GovCloud (US) with AWS DataSync](https://aws.amazon.com/blogs/publicsector/move-data-in-out-aws-govcloud-datasync/) blog.
### Creating an object storage location for your S3 bucket
You need an object storage location for the S3 bucket that's in the Region where you aren't running your DataSync task.
#### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. Make sure that you're in the same Region where you plan to run your task.
1. In the left navigation pane, expand **Data transfer**, then choose **Locations** and **Create location**.
1. For **Location type**, choose **Object storage**.
1. For **Agents**, choose the DataSync agent that you created for this transfer.
1. For **Server**, enter an Amazon S3 endpoint for your bucket by using one of the following formats:
+ **Commercial Region bucket:** `s3.your-region.amazonaws.com`
+ **AWS GovCloud (US) Region bucket**: `s3.your-gov-region.amazonaws.com`
For a list of Amazon S3 endpoints, see the *[AWS General Reference](https://docs.aws.amazon.com/general/latest/gr/s3.html)*.
1. For **Bucket** name, enter the name of the S3 bucket.
1. For **Folder**, enter a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).
**Warning**
DataSync can't transfer objects with a prefix that begins with a slash (`/`) or includes `//`, `/./`, or `/../` patterns. For example:
`/photos`
`photos//2006/January`
`photos/./2006/February`
`photos/../2006/March`
1. Select **Requires credentials** and do the following:
+ For **Access key**, enter the access key for an [IAM user](#create-s3-location-govcloud-iam) that can access the bucket.
+ For **Secret key**, enter the same IAM user’s secret key.
1. (Optional) Choose **Add tag** to tag your location.
Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.
1. Choose **Create location**.
#### Using the AWS CLI
1. Copy the following `create-location-object-storage` command:
```
aws datasync create-location-object-storage \
--server-hostname s3-endpoint \
--bucket-name amzn-s3-demo-bucket \
--agent-arns arn:aws:datasync:your-region:123456789012:agent/agent-01234567890deadfb
```
1. For the `--server-hostname` parameter, specify an Amazon S3 endpoint for your bucket by using one of the following formats:
+ **Commercial Region bucket:** `s3.your-region.amazonaws.com`
+ **AWS GovCloud (US) Region bucket**: `s3.your-gov-region.amazonaws.com`
For the Region in the endpoint, make sure that you specify the same Region where you plan to run your task.
For a list of Amazon S3 endpoints, see the *[AWS General Reference](https://docs.aws.amazon.com/general/latest/gr/s3.html)*.
1. For the `--bucket-name` parameter, specify the name of the S3 bucket.
1. For the `--agent-arns` parameter, specify the DataSync agent that you created for this transfer.
1. For the `--access-key` parameter, specify the access key for an [IAM user](#create-s3-location-govcloud-iam) that can access the bucket.
1. For the `--secret-key` parameter, enter the same IAM user's secret key.
1. (Optional) For the `--subdirectory` parameter, specify a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).
**Warning**
DataSync can't transfer objects with a prefix that begins with a slash (`/`) or includes `//`, `/./`, or `/../` patterns. For example:
`/photos`
`photos//2006/January`
`photos/./2006/February`
`photos/../2006/March`
1. (Optional) For the `--tags` parameter, specify key-value pairs that represent tags for the location resource.
Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.
1. Run the `create-location-object-storage` command.
You get a response that shows you the location ARN that you just created.
```
{
"LocationArn": "arn:aws:datasync:us-east-1:123456789012:location/loc-01234567890abcdef"
}
```
You can use this location as a source or destination for your DataSync task. For the other S3 bucket in this transfer, [create an Amazon S3 location](#create-s3-location-how-to).
## Next steps
Some possible next steps include:
1. If needed, create your other location. For more information, see [Where can I transfer my data with AWS DataSync?](working-with-locations.md)
1. [Configure DataSync task settings](task-options.md), such as what files to transfer, how to handle metadata, among other options.
1. [Set a schedule](task-scheduling.md) for your DataSync task.
1. [Configure monitoring](monitoring-overview.md) for your DataSync task.
1. [Start](run-task.md) your task.
# Configuring AWS DataSync transfers with Amazon EFS
To transfer data to or from your Amazon EFS file system, you must create an AWS DataSync transfer *location*. DataSync can use this location as a source or destination for transferring data.
## Providing DataSync access to Amazon EFS file systems
[Creating a location](#create-efs-location-how-to) involves understanding how DataSync can access your storage. For Amazon EFS, DataSync mounts your file system as a root user from your virtual private cloud (VPC) using [network interfaces](required-network-interfaces.md).
**Contents**
+ [Determining the subnet and security groups for your mount target](#create-efs-location-mount-target)
+ [Accessing restricted file systems](#create-efs-location-iam)
+ [Creating a DataSync IAM role for file system access](#create-efs-location-iam-role)
+ [Example file system policy allowing DataSync access](#create-efs-location-iam-policy)
### Determining the subnet and security groups for your mount target
When creating your location, you specify the subnet and security groups that allow DataSync to connect to one of your Amazon EFS file system's [mount targets](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs.html).
The subnet that you specify must be located:
+ In the same VPC as your file system.
+ In the same Availability Zone as at least one mount target for your file system.
**Note**
You don't need to specify a subnet that includes a file system mount target.
The security groups that you specify must allow inbound traffic on Network File System (NFS) port 2049. For information on creating and updating security groups for your mount targets, see the [https://docs.aws.amazon.com/efs/latest/ug/network-access.html](https://docs.aws.amazon.com/efs/latest/ug/network-access.html).
**Specifying security groups associated with a mount target**
You can specify a security group that's associated with one of your file system's mount targets. We recommend this approach from a network management standpoint.
**Specifying security groups that aren't associated with a mount target**
You also can specify a security group that isn't associated with one of your file system's mount targets. However, this security group must be able to communicate with a mount target's security group.
For example, here's how you might create a relationship between security group D (for DataSync) and security group M (for the mount target):
+ Security group D, which you specify when creating your location, must have a rule that allows outbound connections on NFS port 2049 to security group M.
+ Security group M, which you associate with the mount target, must allow inbound access on NFS port 2049 from security group D.
**To find a mount target's security group**
The following instructions can help you identify the security group of an Amazon EFS file system mount target that you want DataSync to use for your transfer.
1. In the AWS CLI, run the following `describe-mount-targets` command.
```
aws efs describe-mount-targets \
--region file-system-region \
--file-system-id file-system-id
```
This command returns information about your file system's mount targets (similar to the following example output).
```
{
"MountTargets": [
{
"OwnerId": "111222333444",
"MountTargetId": "fsmt-22334a10",
"FileSystemId": "fs-123456ab",
"SubnetId": "subnet-f12a0e34",
"LifeCycleState": "available",
"IpAddress": "11.222.0.123",
"NetworkInterfaceId": "eni-1234a044"
}
]
}
```
1. Take note of the `MountTargetId` value that you want to use.
1. Run the following `describe-mount-target-security-groups` command using the `MountTargetId` to see the security group of your mount target.
```
aws efs describe-mount-target-security-groups \
--region file-system-region \
--mount-target-id mount-target-id
```
You specify this security group when [creating your location](#create-efs-location-how-to).
### Accessing restricted file systems
DataSync can transfer to or from Amazon EFS file systems that restrict access through [access points](https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html) and [IAM policies](https://docs.aws.amazon.com/efs/latest/ug/iam-access-control-nfs-efs.html).
**Note**
If DataSync accesses a destination file system through an access point that [enforces user identity](https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html#enforce-identity-access-points), the POSIX user and group IDs for your source data aren't preserved if you configure your DataSync task to [copy ownership](configure-metadata.md). Instead, the transferred files and folders are set to the access point's user and group IDs. When this happens, task verification fails because DataSync detects a mismatch between metadata in the source and destination locations.
**Contents**
+ [Creating a DataSync IAM role for file system access](#create-efs-location-iam-role)
+ [Example file system policy allowing DataSync access](#create-efs-location-iam-policy)
#### Creating a DataSync IAM role for file system access
If you have an Amazon EFS file system that restricts access through an IAM policy, you can create an IAM role that provides DataSync permission to read from or write data to the file system. You then might need to specify that role in your [file system policy](#create-efs-location-iam-policy).
**To create the DataSync IAM role**
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the left navigation pane, under **Access management**, choose **Roles**, and then choose **Create role**.
1. On the **Select trusted entity** page, for **Trusted entity type**, choose **Custom trust policy**.
1. Paste the following JSON into the policy editor:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": "datasync.amazonaws.com"
},
"Action": "sts:AssumeRole"
}]
}
```
------
1. Choose **Next**. On the **Add permissions** page, choose **Next**.
1. Give your role a name and choose **Create role**.
You specify this role when [creating your location](#create-efs-location-how-to).
#### Example file system policy allowing DataSync access
The following example file system policy shows how access to an Amazon EFS file system (identified in the policy as `fs-1234567890abcdef0`) is restricted but still allows access to DataSync through an IAM role named `MyDataSyncRole`:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Id": "ExampleEFSFileSystemPolicy",
"Statement": [{
"Sid": "AccessEFSFileSystem",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/MyDataSyncRole"
},
"Action": [
"elasticfilesystem:ClientMount",
"elasticfilesystem:ClientWrite",
"elasticfilesystem:ClientRootAccess"
],
"Resource": "arn:aws:elasticfilesystem:us-east-1:111122223333:file-system/fs-1234567890abcdef0",
"Condition": {
"Bool": {
"aws:SecureTransport": "true"
},
"StringEquals": {
"elasticfilesystem:AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:111122223333:access-point/fsap-abcdef01234567890"
}
}
}]
}
```
------
+ `Principal` – Specifies an [IAM role](#create-efs-location-iam) that gives DataSync permission to access the file system.
+ `Action` – Gives DataSync root access and allows it to read from and write to the file system.
+ `aws:SecureTransport` – Requires NFS clients to use TLS when connecting to the file system.
+ `elasticfilesystem:AccessPointArn` – Allows access to the file system only through a specific access point.
## Network considerations with Amazon EFS transfers
VPCs that you use with DataSync must have default tenancy. VPCs with dedicated tenancy aren't supported.
## Performance considerations with Amazon EFS transfers
Your Amazon EFS file system's throughput mode can affect transfer duration and file system performance during the transfer. Consider the following:
+ For best results, we recommend using Elastic throughput mode. If you don't use Elastic throughput mode, your transfer might take longer.
+ If you use Bursting throughput mode, the performance of your file system's applications might be affected because DataSync consumes file system burst credits.
+ How you [configure DataSync to verify your transferred data](configure-data-verification-options.md) can affect file system performance and data access costs.
For more information, see [Amazon EFS performance](https://docs.aws.amazon.com/efs/latest/ug/performance.html) in the *Amazon Elastic File System User Guide* and the [Amazon EFS Pricing](https://aws.amazon.com/efs/pricing/) page.
## Creating your Amazon EFS transfer location
To create the transfer location, you need an existing Amazon EFS file system. If you don't have one, see [Getting started with Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/getting-started.html) in the *Amazon Elastic File System User Guide*.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Locations** and **Create location**.
1. For ** Location type**, choose **Amazon EFS file system**.
You configure this location as a source or destination later.
1. For **File system**, choose the Amazon EFS file system that you want to use as a location.
1. For **Mount path**, enter a mount path for your Amazon EFS file system.
This specifies where DataSync reads or writes data (depending on if this is a source or destination location) on your file system.
By default, DataSync uses the root directory (or [access point](https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html) if you provide one for the **EFS access point** setting). You can also specify subdirectories using forward slashes (for example, `/path/to/directory`).
1. For **Subnet** choose a subnet where you want DataSync to create the [network interfaces](required-network-interfaces.md) for managing your data transfer traffic.
The subnet must be located:
+ In the same VPC as your file system.
+ In the same Availability Zone as at least one file system mount target.
**Note**
You don't need to specify a subnet that includes a file system mount target.
1. For **Security groups**, choose the security group associated with your Amazon EFS file system's mount target. You can choose more than one security group.
**Note**
The security groups that you specify must allow inbound traffic on NFS port 2049. For more information, see [Determining the subnet and security groups for your mount target](#create-efs-location-mount-target).
1. For **In-transit encryption**, choose whether you want DataSync to use Transport Layer Security (TLS) encryption when it transfers data to or from your file system.
**Note**
You must enable this setting to configure an access point, IAM role, or both with your Amazon EFS location.
1. (Optional) For **EFS access point**, choose an access point that DataSync can use to mount your file system.
For more information, see [Accessing restricted file systems](#create-efs-location-iam).
1. (Optional) For **IAM role**, specify a role that allows DataSync to access your file system.
For information on creating this role, see [Creating a DataSync IAM role for file system access](#create-efs-location-iam-role).
1. (Optional) Select **Add tag** to tag your file system.
A *tag* is a key-value pair that helps you manage, filter, and search for your locations.
1. Choose **Create location**.
### Using the AWS CLI
1. Copy the following `create-location-efs` command:
```
aws datasync create-location-efs \
--efs-filesystem-arn 'arn:aws:elasticfilesystem:region:account-id:file-system/file-system-id' \
--subdirectory /path/to/your/subdirectory \
--ec2-config SecurityGroupArns='arn:aws:ec2:region:account-id:security-group/security-group-id',SubnetArn='arn:aws:ec2:region:account-id:subnet/subnet-id' \
--in-transit-encryption TLS1_2 \
--access-point-arn 'arn:aws:elasticfilesystem:region:account-id:access-point/access-point-id' \
--file-system-access-role-arn 'arn:aws:iam::account-id:role/datasync-efs-access-role
```
1. For `--efs-filesystem-arn`, specify the Amazon Resource Name (ARN) of the Amazon EFS file system that you're transferring to or from.
1. For `--subdirectory`, specify a mount path for your file system.
This is where DataSync reads or writes data (depending on if this is a source or destination location) on your file system.
By default, DataSync uses the root directory (or [access point](https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html) if you provide one with `--access-point-arn`). You can also specify subdirectories using forward slashes (for example, `/path/to/directory`).
1. For `--ec2-config`, do the following:
+ For `SecurityGroupArns`, specify the ARN of the security group associated with your file system's mount target. You can specify more than one security group.
**Note**
The security groups that you specify must allow inbound traffic on NFS port 2049. For more information, see [Determining the subnet and security groups for your mount target](#create-efs-location-mount-target).
+ For `SubnetArn`, specify the ARN of the subnet where you want DataSync to create the [network interfaces](required-network-interfaces.md) for managing your data transfer traffic.
The subnet must be located:
+ In the same VPC as your file system.
+ In the same Availability Zone as at least one file system mount target.
**Note**
You don't need to specify a subnet that includes a file system mount target.
1. For `--in-transit-encryption`, specify whether you want DataSync to use Transport Layer Security (TLS) encryption when it transfers data to or from your file system.
**Note**
You must set this to `TLS1_2` to configure an access point, IAM role, or both with your Amazon EFS location.
1. (Optional) For `--access-point-arn`, specify the ARN of an access point that DataSync can use to mount your file system.
For more information, see [Accessing restricted file systems](#create-efs-location-iam).
1. (Optional) For `--file-system-access-role-arn`, specify the ARN of an IAM role that allows DataSync to access your file system.
For information on creating this role, see [Creating a DataSync IAM role for file system access](#create-efs-location-iam-role).
1. Run the `create-location-efs` command.
If the command is successful, you get a response that shows you the ARN of the location that you created. For example:
```
{
"LocationArn": "arn:aws:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d"
}
```
# Configuring transfers with FSx for Windows File Server
To transfer data to or from your Amazon FSx for Windows File Server file system, you must create an AWS DataSync transfer *location*. DataSync can use this location as a source or destination for transferring data.
## Providing DataSync access to FSx for Windows File Server file systems
DataSync connects to your FSx for Windows File Server file system with the Server Message Block (SMB) protocol and mounts it from your virtual private cloud (VPC) using [network interfaces](required-network-interfaces.md).
**Note**
VPCs that you use with DataSync must have default tenancy. VPCs with dedicated tenancy aren't supported.
**Topics**
+ [Required permissions](#create-fsx-windows-location-permissions)
+ [Required authentication protocols](#configuring-fsx-windows-authentication-protocols)
+ [DFS Namespaces](#configuring-fsx-windows-location-dfs)
### Required permissions
You must provide DataSync a user with the necessary rights to mount and access your FSx for Windows File Server files, folders, and file metadata.
We recommend that this user belong to a Microsoft Active Directory group for administering your file system. The specifics of this group depends on your Active Directory setup:
+ If you're using AWS Directory Service for Microsoft Active Directory with FSx for Windows File Server, the user must be a member of the **AWS Delegated FSx Administrators** group.
+ If you're using self-managed Active Directory with FSx for Windows File Server, the user must be a member of one of two groups:
+ The **Domain Admins** group, which is the default delegated administrators group.
+ A custom delegated administrators group with user rights that allow DataSync to copy object ownership permissions and Windows access control lists (ACLs).
**Important**
You can't change the delegated administrators group after the file system has been deployed. You must either redeploy the file system or restore it from a backup to use the custom delegated administrator group with the following user rights that DataSync needs to copy metadata.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/create-fsx-location.html)
+ If you want to copy Windows ACLs and are transferring between an SMB file server and FSx for Windows File Server file system or between FSx for Windows File Server file systems, the users that you provide DataSync must belong to the same Active Directory domain or have an Active Directory trust relationship between their domains.
**Warning**
Your FSx for Windows File Server file system's SYSTEM user must have **Full control** permissions on all folders in your file system. Do not change the NTFS ACL permissions for this user on your folders. If you do, DataSync can change your file system's permissions in a way that makes your file share inaccessible and prevents file system backups from being usable. For more information on file- and folder-level access, see the*[ Amazon FSx for Windows File Server User Guide](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/limit-access-file-folder.html)*.
### Required authentication protocols
Your FSx for Windows File Server must use NTLM authentication for DataSync to access it. DataSync can't access a file server that uses Kerberos authentication.
### DFS Namespaces
DataSync doesn't support Microsoft Distributed File System (DFS) Namespaces. We recommend specifying an underlying file server or share instead when creating your DataSync location.
For more information, see [Grouping multiple file systems with DFS Namespaces](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/group-file-systems.html) in the *Amazon FSx for Windows File Server User Guide*.
## Creating your FSx for Windows File Server transfer location
Before you begin, make sure that you have an existing FSx for Windows File Server in your AWS Region. For more information, see [Getting started with Amazon FSx ](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/getting-started.html) in the *Amazon FSx for Windows File Server User Guide*.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Locations** and **Create location**.
1. For **Location type**, choose **Amazon FSx**.
1. For **FSx file system**, choose the FSx for Windows File Server file system that you want to use as a location.
1. For **Share name**, enter a mount path for your FSx for Windows File Server using forward slashes.
This specifies the path where DataSync reads or writes data (depending on if this is a source or destination location).
You can also include subdirectories (for example, `/path/to/directory`).
1. For **Security groups**, choose up to five Amazon EC2 security groups that provide access to your file system's preferred subnet.
The security groups that you choose must be able to communicate with your file system's security groups. For information about configuring security groups for file system access, see the [https://docs.aws.amazon.com/fsx/latest/WindowsGuide/limit-access-security-groups.html](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/limit-access-security-groups.html).
**Note**
If you choose a security group that doesn't allow connections from within itself, do one of the following:
Configure the security group to allow it to communicate within itself.
Choose a different security group that can communicate with the mount target's security group.
1. For **User**, enter the name of a user that can access your FSx for Windows File Server.
For more information, see [Required permissions](#create-fsx-windows-location-permissions).
1. For **Password**, enter password of the user name.
1. (Optional) For **Domain**, enter the name of the Windows domain that your FSx for Windows File Server file system belongs to.
If you have multiple Active Directory domains in your environment, configuring this setting makes sure that DataSync connects to the right file system.
1. (Optional) Enter values for the **Key** and **Value** fields to tag the FSx for Windows File Server.
Tags help you manage, filter, and search for your AWS resources. We recommend creating at least a name tag for your location.
1. Choose **Create location**.
### Using the AWS CLI
**To create an FSx for Windows File Server location by using the AWS CLI**
+ Use the following command to create an Amazon FSx location.
```
aws datasync create-location-fsx-windows \
--fsx-filesystem-arn arn:aws:fsx:region:account-id:file-system/filesystem-id \
--security-group-arns arn:aws:ec2:region:account-id:security-group/group-id \
--user smb-user --password password
```
In the `create-location-fsx-windows` command, do the following:
+ `fsx-filesystem-arn` – Specify the Amazon Resource Name (ARN) of the file system that you want to transfer to or from.
+ `security-group-arns` – Specify the ARNs of up to five Amazon EC2 security groups that provide access to your file system's preferred subnet.
The security groups that you specify must be able to communicate with your file system's security groups. For information about configuring security groups for file system access, see the [https://docs.aws.amazon.com/fsx/latest/WindowsGuide/limit-access-security-groups.html](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/limit-access-security-groups.html).
**Note**
If you choose a security group that doesn't allow connections from within itself, do one of the following:
Configure the security group to allow it to communicate within itself.
Choose a different security group that can communicate with the mount target's security group.
+ The AWS Region – The Region that you specify is the one where your target Amazon FSx file system is located.
The preceding command returns a location ARN similar to the one shown following.
```
{
"LocationArn": "arn:aws:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb"
}
```
# Configuring DataSync transfers with FSx for Lustre
To transfer data to or from your Amazon FSx for Lustre file system, you must create an AWS DataSync transfer *location*. DataSync can use this location as a source or destination for transferring data.
## Providing DataSync access to FSx for Lustre file systems
DataSync accesses your FSx for Lustre file system using the Lustre client. DataSync requires access to all data on your FSx for Lustre file system. To have this level of access, DataSync mounts your file system as the root user using a user ID (UID) and group ID (GID) of `0`.
DataSync mounts your file system from your virtual private cloud (VPC) using [network interfaces](required-network-interfaces.md). DataSync fully manages the creation, the use, and the deletion of these network interfaces on your behalf.
**Note**
VPCs that you use with DataSync must have default tenancy. VPCs with dedicated tenancy aren't supported.
## Creating your FSx for Lustre transfer location
To create the transfer location, you need an existing FSx for Lustre file system. For more information, see [Getting started with Amazon FSx for Lustre](https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started.html) in the *Amazon FSx for Lustre User Guide*.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Locations** and **Create location**.
1. For **Location type**, choose **Amazon FSx**.
You configure this location as a source or destination later.
1. For **FSx file system**, choose the FSx for Lustre file system that you want to use as a location.
1. For **Mount path**, enter the mount path for your FSx for Lustre file system.
The path can include a subdirectory. When the location is used as a source, DataSync reads data from the mount path. When the location is used as a destination, DataSync writes all data to the mount path. If a subdirectory isn't provided, DataSync uses the root directory (`/`).
1. For **Security groups**, choose up to five security groups that provide access to your FSx for Lustre file system.
The security groups must be able to access the file system's ports. The file system must also allow access from the security groups.
For more information about security groups, see [File System Access Control with Amazon VPC](https://docs.aws.amazon.com/fsx/latest/LustreGuide/limit-access-security-groups.html) in the *Amazon FSx for Lustre User Guide*.
1. (Optional) Enter values for the **Key** and **Value** fields to tag the FSx for Lustre file system.
Tags help you manage, filter, and search for your AWS resources. We recommend creating at least a name tag for your location.
1. Choose **Create location**.
### Using the AWS CLI
**To create an FSx for Lustre location by using the AWS CLI**
+ Use the following command to create an FSx for Lustre location.
```
aws datasync create-location-fsx-lustre \
--fsx-filesystem-arn arn:aws:fsx:region:account-id:file-system:filesystem-id \
--security-group-arns arn:aws:ec2:region:account-id:security-group/group-id
```
The following parameters are required in the `create-location-fsx-lustre` command.
+ `fsx-filesystem-arn` – The fully qualified Amazon Resource Name (ARN) of the file system that you want to read from or write to.
+ `security-group-arns` – The ARN of an Amazon EC2 security group to apply to the [network interfaces](required-network-interfaces.md) of the file system's preferred subnet.
The preceding command returns a location ARN similar to the following.
```
{
"LocationArn": "arn:aws:datasync:us-west-2:111222333444:location/loc-07sb7abfc326c50fb"
}
```
# Configuring DataSync transfers with Amazon FSx for OpenZFS
To transfer data to or from your Amazon FSx for OpenZFS file system, you must create an AWS DataSync transfer *location*. DataSync can use this location as a source or destination for transferring data.
## Providing DataSync access to FSx for OpenZFS file systems
DataSync mounts your FSx for OpenZFS file system from your virtual private cloud (VPC) using [network interfaces](required-network-interfaces.md). DataSync fully manages the creation, the use, and the deletion of these network interfaces on your behalf.
**Note**
VPCs that you use with DataSync must have default tenancy. VPCs with dedicated tenancy aren't supported.
## Configuring FSx for OpenZFS file system authorization
DataSync accesses your FSx for OpenZFS file system as an NFS client, mounting the file system as a root user with a user ID (UID) and group ID (GID) of `0`.
For DataSync to copy all of your file metadata, you must configure the NFS export settings on your file system volumes using `no_root_squash`. However, you can limit this level of access to only a specific DataSync task.
For more information, see [Volume properties](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/managing-volumes.html#volume-properties) in the *Amazon FSx for OpenZFS User Guide*.
### Configuring NFS exports specific to DataSync (recommended)
You can configure an NFS export specific to each volume that’s accessed only by your DataSync task. Do this for the most recent ancestor volume of the mount path that you specify when creating your FSx for OpenZFS location.
**To configure an NFS export specific to DataSync**
1. Create your [DataSync task](create-task-how-to.md).
This creates the task’s network interfaces that you specify in your NFS export settings.
1. Locate the private IP addresses of the task's network interfaces by using the Amazon EC2 console or AWS CLI.
1. For your FSx for OpenZFS file system volume, configure the following NFS export settings for each of the task’s network interfaces:
+ **Client address**: Enter the network interface’s private IP address (for example, `10.24.34.0`).
+ **NFS options**: Enter `rw,no_root_squash`.
### Configuring NFS exports for all clients
You can specify an NFS export that allows root access to all clients.
**To configure an NFS export for all clients**
+ For your FSx for OpenZFS file system volume, configure the following NFS export settings:
+ **Client address**: Enter `*`.
+ **NFS options**: Enter `rw,no_root_squash`.
## Creating your FSx for OpenZFS transfer location
To create the location, you need an existing FSx for OpenZFS file system. If you don't have one, see [Getting started with Amazon FSx for OpenZFS](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/getting-started.html) in the *Amazon FSx for OpenZFS User Guide*.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, choose **Locations**, and then choose **Create location**.
1. For **Location type**, choose **Amazon FSx**.
You configure this location as a source or destination later.
1. For **FSx file system**, choose the FSx for OpenZFS file system that you want to use as a location.
1. For **Mount path**, enter the mount path for your FSx for OpenZFS file system.
The path must begin with `/fsx` and can be any existing directory path in the file system. When the location is used as a source, DataSync reads data from the mount path. When the location is used as a destination, DataSync writes all data to the mount path. If a subdirectory isn't provided, DataSync uses the root volume directory (for example, `/fsx`).
1. For **Security groups**, choose up to five security groups that provide network access to your FSx for OpenZFS file system.
The security groups must provide access to the network ports that are used by the FSx for OpenZFS file system. The file system must allow network access from the security groups.
For more information about security groups, see [File system access control with Amazon VPC](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/limit-access-security-groups.html) in the *Amazon FSx for OpenZFS User Guide*.
1. (Optional) Expand **Additional settings** and for **NFS version** choose the NFS version that DataSync uses to access your file system.
By default, DataSync uses NFS version 4.1.
1. (Optional) Enter values for the **Key** and **Value** fields to tag the FSx for OpenZFS file system.
Tags help you manage, filter, and search for your location. We recommend creating at least a name tag for your location.
1. Choose **Create location**.
### Using the AWS CLI
**To create an FSx for OpenZFS location by using the AWS CLI**
1. Copy the following `create-location-fsx-open-zfs` command:
```
aws datasync create-location-fsx-open-zfs \
--fsx-filesystem-arn arn:aws:fsx:region:account-id:file-system/filesystem-id \
--security-group-arns arn:aws:ec2:region:account-id:security-group/group-id \
--protocol NFS={}
```
1. Specify the following required options in the command:
+ For `fsx-filesystem-arn`, specify the location file system's fully qualified Amazon Resource Name (ARN). This includes the AWS Region where your file system resides, your AWS account, and the file system ID.
+ For `security-group-arns`, specify the ARN of the Amazon EC2 security group that provides access to the [network interfaces](required-network-interfaces.md) of your FSx for OpenZFS file system's preferred subnet. This includes the AWS Region where your Amazon EC2 instance resides, your AWS account, and the security group ID.
For more information about security groups, see [File System Access Control with Amazon VPC](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/limit-access-security-groups.html) in the *Amazon FSx for OpenZFS User Guide*.
+ For `protocol`, specify the protocol that DataSync uses to access your file system. (DataSync currently supports only NFS.)
1. Run the command. You get a response showing the location that you just created.
```
{
"LocationArn": "arn:aws:datasync:us-west-2:123456789012:location/loc-abcdef01234567890"
}
```
# Configuring transfers with Amazon FSx for NetApp ONTAP
To transfer data to or from your Amazon FSx for NetApp ONTAP file system, you must create an AWS DataSync transfer *location*. DataSync can use this location as a source or destination for transferring data.
## Providing DataSync access to FSx for ONTAP file systems
To access an FSx for ONTAP file system, DataSync mounts a storage virtual machine (SVM) on your file system using [network interfaces](required-network-interfaces.md) in your virtual private cloud (VPC). DataSync creates these network interfaces in your file system’s preferred subnet only when you create a task that includes your FSx for ONTAP location.
**Note**
VPCs that you use with DataSync must have default tenancy. VPCs with dedicated tenancy aren't supported.
DataSync can connect to an FSx for ONTAP file system's SVM and copy data by using the Network File System (NFS) or Server Message Block (SMB) protocol.
**Topics**
+ [Using the NFS protocol](#create-ontap-location-supported-protocols)
+ [Using the SMB protocol](#create-ontap-location-smb)
+ [Unsupported protocols](#create-ontap-location-unsupported-protocols)
+ [Choosing the right protocol](#create-ontap-location-choosing-protocol)
+ [Accessing SnapLock volumes](#create-ontap-location-snaplock)
### Using the NFS protocol
With the NFS protocol, DataSync uses the `AUTH_SYS` security mechanism with a user ID (UID) and group ID (GID) of `0` to authenticate with your SVM.
**Note**
DataSync currently only supports NFS version 3 with FSx for ONTAP locations.
### Using the SMB protocol
With the SMB protocol, DataSync uses credentials that you provide to authenticate with your SVM.
**Supported SMB versions**
By default, DataSync automatically chooses a version of the SMB protocol based on negotiation with your SMB file server. You also can configure DataSync to use a specific version, but we recommend doing this only if DataSync has trouble negotiating with the SMB file server automatically. For security reasons, we recommend using SMB version 3.0.2 or later.
See the following table for a list of options in the DataSync console and API for configuring an SMB version with your FSx for ONTAP location:
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/create-ontap-location.html)
**Required permissions**
You must provide DataSync a local user in your SVM or a domain user in your Microsoft Active Directory with the necessary rights to mount and access your files, folders, and file metadata.
If you provide a user in your Active Directory, note the following:
+ If you're using AWS Directory Service for Microsoft Active Directory, the user must be a member of the **AWS Delegated FSx Administrators** group.
+ If you're using a self-managed Active Directory, the user must be a member of one of two groups:
+ The **Domain Admins** group, which is the default delegated administrators group.
+ A custom delegated administrators group with user rights that allow DataSync to copy object ownership permissions and Windows access control lists (ACLs).
**Important**
You can't change the delegated administrators group after the file system has been deployed. You must either redeploy the file system or restore it from a backup to use the custom delegated administrator group with the following user rights that DataSync needs to copy metadata.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/create-ontap-location.html)
+ If you want to copy Windows ACLs and are transferring between FSx for ONTAP file systems using SMB (or other types of file systems using SMB), the users that you provide DataSync must belong to the same Active Directory domain or have an Active Directory trust relationship between their domains.
**Required authentication protocols**
For DataSync to access your SMB share, your FSx for ONTAP file system must use NTLM authentication. DataSync can't access FSx for ONTAP file systems that use Kerberos authentication.
**DFS Namespaces**
DataSync doesn't support Microsoft Distributed File System (DFS) Namespaces. We recommend specifying an underlying file server or share instead when creating your DataSync location.
### Unsupported protocols
DataSync can't access FSx for ONTAP file systems using the iSCSI (Internet Small Computer Systems Interface) protocol.
### Choosing the right protocol
To preserve file metadata in FSx for ONTAP migrations, configure your DataSync source and destination locations to use the same protocol. Between the supported protocols, SMB preserves metadata with the highest fidelity (see [Understanding how DataSync handles file and object metadata](metadata-copied.md) for details).
When migrating from a Unix (Linux) server or network-attached storage (NAS) share that serves users through NFS, do the following:
1. [Create an NFS location](create-nfs-location.md) for the Unix (Linux) server or NAS share. (This will be your source location.)
1. Configure the FSx for ONTAP volume you’re transferring data to with the [Unix security style](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/managing-volumes.html#volume-security-style).
1. Create a location for your FSx for ONTAP file system that’s configured for NFS. (This will be your destination location.)
When migrating from a Windows server or NAS share that serves users through SMB, do the following:
1. [Create an SMB location](create-smb-location.md) for the Windows server or NAS share. (This will be your source location.)
1. Configure the FSx for ONTAP volume you’re transferring data to with the [NTFS security style](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/managing-volumes.html#volume-security-style).
1. Create a location for your FSx for ONTAP file system that’s configured for SMB. (This will be your destination location.)
If your FSx for ONTAP environment uses multiple protocols, we recommend working with an AWS storage specialist. To learn about best practices for multiprotocol access, see [Enabling multiprotocol workloads with Amazon FSx for NetApp ONTAP](https://aws.amazon.com/blogs/storage/enabling-multiprotocol-workloads-with-amazon-fsx-for-netapp-ontap/).
### Accessing SnapLock volumes
If you're transferring data to a [SnapLock volume](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/snaplock.html) on an FSx for ONTAP file system, make sure the SnapLock settings **Autocommit** and **Volume append mode** are disabled on the volume during your transfer. You can re-enable these settings when you're done transferring data.
## Creating your FSx for ONTAP transfer location
To create the location, you need an existing FSx for ONTAP file system. If you don't have one, see [Getting started with Amazon FSx for NetApp ONTAP](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/getting-started.html) in the *Amazon FSx for NetApp ONTAP User Guide*.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Locations** and **Create location**.
1. For **Location type**, choose **Amazon FSx**.
You configure this location as a source or destination later.
1. For **FSx file system**, choose the FSx for ONTAP file system that you want to use as a location.
1. For **Storage virtual machine**, choose a storage virtual machine (SVM) in your file system where you want to copy data to or from.
1. For **Mount path**, specify a path to the file share in that SVM where you'll copy your data.
You can specify a junction path (also known as a mount point), qtree path (for NFS file shares), or share name (for SMB file shares). For example, your mount path might be `/vol1`, `/vol1/tree1`, or `/share1`.
**Tip**
Don't specify a path in the SVM's root volume. For more information, see [Managing FSx for ONTAP storage virtual machines](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/managing-svms.html) in the *Amazon FSx for NetApp ONTAP User Guide*.
1. For **Security groups**, choose up to five Amazon EC2 security groups that provide access to your file system's preferred subnet.
The security groups must allow outbound traffic on the following ports (depending on the protocol you use):
+ **NFS** – TCP ports 111, 635, and 2049
+ **SMB** – TCP port 445
Your file system's security groups must also allow inbound traffic on the same ports.
1. For **Protocol**, choose the data transfer protocol that DataSync uses to access your file system's SVM.
For more information, see [Choosing the right protocol](#create-ontap-location-choosing-protocol).
------
#### [ NFS ]
DataSync uses NFS version 3.
------
#### [ SMB ]
Configure an SMB version, user, password, and Active Directory domain name (if needed) to access the SVM.
+ (Optional) Expand **Additional settings** and choose an **SMB version** for DataSync to use when accessing your SVM.
By default, DataSync automatically chooses a version based on negotiation with the SMB file server. For more information, see [Using the SMB protocol](#create-ontap-location-smb).
+ For **User**, enter a user name that can mount and access the files, folders, and metadata that you want to transfer in the SVM.
For more information, see [Using the SMB protocol](#create-ontap-location-smb).
+ For **Password**, enter the password of the user that you specified that can access the SVM.
+ (Optional) For **Active Directory domain name**, enter the fully qualified domain name (FQDN) of the Active Directory that your SVM belongs to.
If you have multiple domains in your environment, configuring this setting makes sure that DataSync connects to the right SVM.
------
1. (Optional) Enter values for the **Key** and **Value** fields to tag the FSx for ONTAP file system.
Tags help you manage, filter, and search for your AWS resources. We recommend creating at least a name tag for your location.
1. Choose **Create location**.
### Using the AWS CLI
**To create an FSx for ONTAP location by using the AWS CLI**
1. Copy the following `create-location-fsx-ontap` command:
```
aws datasync create-location-fsx-ontap \
--storage-virtual-machine-arn arn:aws:fsx:region:account-id:storage-virtual-machine/fs-file-system-id \
--security-group-arns arn:aws:ec2:region:account-id:security-group/group-id \
--protocol data-transfer-protocol={}
```
1. Specify the following required options in the command:
+ For `storage-virtual-machine-arn`, specify the fully qualified Amazon Resource Name (ARN) of a storage virtual machine (SVM) in your file system where you want to copy data to or from.
This ARN includes the AWS Region where your file system resides, your AWS account, and the file system and SVM IDs.
+ For `security-group-arns`, specify the ARNs of the Amazon EC2 security groups that provide access to the [network interfaces](required-network-interfaces.md) of your file system's preferred subnet.
This includes the AWS Region where your Amazon EC2 instance resides, your AWS account, and your security group IDs. You can specify up to five security group ARNs.
For more information about security groups, see [File System Access Control with Amazon VPC](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/limit-access-security-groups.html) in the *Amazon FSx for NetApp ONTAP User Guide*.
+ For `protocol`, configure the protocol that DataSync uses to access your file system's SVM.
+ For NFS, you can use the default configuration:
`--protocol NFS={}`
+ For SMB, you must specify a user name and password that can access the SVM:
`--protocol SMB={User=smb-user,Password=smb-password}`
1. Run the command.
You get a response that shows the location that you just created.
```
{
"LocationArn": "arn:aws:datasync:us-west-2:123456789012:location/loc-abcdef01234567890"
}
```