# Troubleshooting AWS DataSync issues Use the following information to troubleshoot AWS DataSync issues and errors. **Topics** + [Troubleshooting issues with DataSync agents](troubleshooting-datasync-agents.md) + [Troubleshooting issues with DataSync locations](troubleshooting-storage-issues.md) + [Troubleshooting issues with DataSync tasks](troubleshooting-tasks.md) + [Troubleshooting data verification issues](troubleshooting-task-verification.md) + [Troubleshooting higher than expected S3 storage costs with DataSync](multipart-upload-policy.md) # Troubleshooting issues with DataSync agents Use the following information to help you troubleshoot issues with AWS DataSync agents. Some of these issues can include: + Trouble connecting to an Amazon EC2 agent's local console + Failing to retrieve an agent's activation key + Issues activating an agent with a VPC service endpoint + Discovering an agent is offline ## How do I connect to an Amazon EC2 agent's local console? To connect to an Amazon EC2 agent's local console, you must use SSH. Make sure that your EC2 instance's security group allows access with SSH (TCP port 22). In a terminal, run the following `ssh` command to connect to the instance: ``` ssh -i /path/key-pair-name.pem instance-user-name@instance-public-ip-address ``` + For */path/key-pair-name*, specify the path and file name (`.pem`) of the private key required to connect to your instance. + For *instance-user-name*, specify `admin`. + For *instance-public-ip-address*, specify the public IP address of your instance. ## What does the Failed to retrieve agent activation key error mean? When activating your DataSync agent, the agent connects to the service endpoint that you specify to request an activation key. This error likely means that your network security settings are blocking the connection. **Action to take** If you're using a virtual private cloud (VPC) service endpoint, verify that your security group settings allow your agent to connect to the VPC endpoint. For information about required ports, see [Network requirements for VPC or FIPS VPC service endpoints](datasync-network.md#using-vpc-endpoint). If you're using a public or Federal Information Processing Standard (FIPS) endpoint, check that your firewall and router settings allow your agent to connect to the endpoint. For information, see [Network requirements for public or FIPS service endpoints](datasync-network.md#using-public-endpoints). ## I still can't activate an agent by using a VPC service endpoint If you're still having issues activating a DataSync agent with a VPC service endpoint, see [I don't know what's going on with my agent. Can someone help me?](#enable-support-access) ## What do I do if my agent is offline? Your DataSync agent can be offline for a few reasons, but you might be able to get it back online. Before you delete the agent and create a new one, go through the following checklist to help you understand what might have happened. + **Contact your backup team** – If your agent is offline because its virtual machine (VM) was restored from a snapshot or backup, you might need to [replace the agent](replacing-agent.md). + **Check if the agent's VM or Amazon EC2 instance is off** – Depending on the type of agent that you're using, try turning the VM or EC2 instance back on if it's off. Once it's on again, [test your agent's network connectivity](test-agent-connections.md#test-network) to AWS. + **Verify your agent meets the minimum hardware requirements** – Your agent might be offline because its VM or EC2 instance configuration was accidentally changed since the agent was activated. For example, if your VM no longer has the minimum required memory or space, the agent might appear as offline. For more information, see [Requirements for AWS DataSync agents](agent-requirements.md). + **Wait for agent-related software updates to finish** – Your agent might go offline briefly following [software updates provided by AWS](managing-agent.md#managing-agent-updates). If you believe this is why the agent is offline, wait a short period then check if the agent is back online. + **Check your VPC service endpoint settings** – If your offline agent is using a public service endpoint and also in the same VPC where you created a VPC service endpoint for DataSync, you might need to disable [private DNS support](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html) for that VPC endpoint. If none of these seem to be the reason that the agent is offline, you likely need to [replace the agent](replacing-agent.md). ## I don't know what's going on with my agent. Can someone help me? You can allow AWS Support to access your DataSync agent and help troubleshoot agent-related issues. You must enable this access through the agent's local console. **To provide Support access to your agent** 1. [Log in to your agent's local console](local-console-vm.md#local-console-login). 1. At the prompt, enter **5** to open the command prompt (for VMware VMs, use **6**). 1. Enter **h** to open the **AVAILABLE COMMANDS** window. 1. In the **AVAILABLE COMMANDS** window, enter the following to connect to Support: `open-support-channel` If you are using the agent with VPC endpoints, you must provide a VPC endpoint IP address for your support channel, as follows: `open-support-channel vpc-ip-address` Your firewall must allow the inbound TCP port 22 to initiate a support channel to AWS. When you connect to Support, DataSync assigns you a support number. Make a note of your support number. **Note** The channel number isn't a TCP/UDP port number. Instead, it makes an SSH (TCP 22) connection to servers and provides the support channel for the connection. 1. When the support channel is established, provide your support service number to Support so that they can provide troubleshooting assistance. 1. When the support session is finished, press **Enter** to end it. 1. Enter **exit** to log out of the DataSync local console. 1. Follow the prompts to exit the local console. # Troubleshooting issues with DataSync locations Use the following information to help you troubleshoot issues with AWS DataSync locations. Some of these issues can include: + Permissions and mount errors with NFS locations + File ownership issues + Problems accessing SMB locations that use Kerberos authentication + Permission and access issues with object storage, such as Amazon S3 and Microsoft Azure Blob locations ## My task failed with an NFS permissions denied error You can get a "permissions denied" error message if you configure your NFS file server with `root_squash` or `all_squash` and your files don't all have read access. **Action to take** To fix this issue, configure your NFS export with `no_root_squash` or make sure that the permissions for all of the files that you want to transfer allow read access for all users. For DataSync to access directories, you must also enable all-execute access. To make sure that the directory can be mounted, first connect to any computer that has the same network configuration as your agent. Then run the following CLI command: `mount -t nfs -o nfsvers= : ` If the issue still isn't resolved, contact [AWS Support Center](https://console.aws.amazon.com/support/home#/). ## My task failed with an NFS mount error You might see the following error when running a DataSync task that involves an NFS file server location: Task failed to access location loc-1111222233334444a: x40016: mount.nfs: Connection timed out **Actions to take** Do the following until the error is resolved. 1. Make sure that the NFS file server and export that you specify in your DataSync location are valid. If they aren't, delete your location and task, then create a new location and task that use a valid NFS file server and export. For more information, see [Using the DataSync console](create-nfs-location.md#create-nfs-location-console). 1. Check your firewall configuration between your agent and NFS file server. For more information, see [Network requirements for on-premises, self-managed, and other cloud storage](datasync-network.md#on-premises-network-requirements). 1. Make sure that your agent can access the NFS file server and mount the export. For more information, see [Providing DataSync access to NFS file servers](create-nfs-location.md#accessing-nfs). 1. If you still see the error, open a support channel with Support. For more information, see [I don't know what's going on with my agent. Can someone help me?](troubleshooting-datasync-agents.md#enable-support-access). ## My task failed with an Amazon EFS mount error You might see the following error when running a DataSync task that involves an Amazon EFS location: Task failed to access location loc-1111222233334444a: x40016: Failed to connect to EFS mount target with IP: 10.10.1.0. This can happen if the Amazon EFS file system's mount path that you configure with your location gets updated or deleted. DataSync isn't aware of these changes in the file system. **Action to take** Delete your location and task and [create a new Amazon EFS location](create-efs-location.md#create-efs-location-how-to) with the new mount path. ## File ownership isn't maintained with NFS transfer After your transfer, you might notice that the files in your DataSync destination location have different user IDs (UIDs) or group IDs (GIDs) than the same files in your source location. For example, the files in your destination might have a UID of `65534`, `99`, or `nobody`. This can happen if a file system involved in your transfer uses NFS version 4 ID mapping, a feature that DataSync doesn't support. **Action to take** You have a couple options to work around this issue: + Create a new location for the file system that uses NFS version 3 instead of version 4. + Disable NFS version 4 ID mapping on the file system. Retry the transfer. Either option should resolve the issue. ## My task can't access an SMB location that uses Kerberos DataSync errors with SMB locations that use [Kerberos authentication](create-smb-location.md#configuring-smb-kerberos-authentication) are typically related to mismatches between your location and Kerberos configurations. There also might be a network issue. **Failed to access location** The following error indicates that there might be configuration issues with your SMB location or Kerberos setup: ``` Task failed to access location ``` **Verify the following**: + The SMB file server that you specify for your location is a domain name. For Kerberos, you can't specify the file server's IP address. + The Kerberos principal that you specify for your location matches the principal that you use to create the Kerberos key table (keytab) file. Principal names are case sensitive. + The Kerberos principal's mapped user password hasn't changed since you created the keytab file. If the password changes (because of password rotation or some other reason), your task execution might fail with the following error: Task failed to access location loc-1111222233334444a: x40015: kinit: Preauthentication failed while getting initial credentials **Can't contact KDC realm** The following error indicates a networking issue: ``` kinit: Cannot contact any KDC for realm 'MYDOMAIN.ORG' while getting initial credentials" ``` **Verify the following**: + The Kerberos configuration file (`krb5.conf`) that you provided DataSync has the correct information about your Kerberos realm. For an example `krb5.conf` file, see [Kerberos authentication prerequisites](create-smb-location.md#configuring-smb-kerberos-prerequisites). + The Kerberos Key Distribution Center (KDC) server port is open. The KDC port is typically TCP port 88. + The DNS configuration on your network. ## My task failed with an input/output error You can get an input/output error message if your storage system fails I/O requests from the DataSync agent. Common reasons for this include a server disk failure, changes to your firewall configuration, or a network router failure. If the error involves an NFS file server or Hadoop Distributed File System (HDFS) cluster, use the following steps to resolve the error. **Actions to take (NFS)** First, check your NFS file server's logs and metrics to determine if the problem started on the NFS server. If yes, resolve that issue. Next, check that your network configuration hasn't changed. To check if the NFS file server is configured correctly and that DataSync can access it, do the following: 1. Set up another NFS client on the same network subnet as the agent. 1. Mount your share on that client. 1. Validate that the client can read and write to the share successfully. **Actions to take (HDFS)** Do the following until you resolve the error: 1. Make sure that your HDFS cluster allows your DataSync agent to communicate with the cluster's NameNode and DataNode ports. In most clusters, you can find the port numbers that the cluster uses in the following configuration files: + To find the NameNode port, look in the `core-site.xml` file under the `fs.default` or `fs.default.name` property (depending on the Hadoop distribution). + To find the DataNode port, look in the `hdfs-site.xml` file under the `dfs.datanode.address` property. 1. In your `hdfs-site.xml` file, verify that your `dfs.data.transfer.protection` property has only one value. For example: ``` dfs.data.transfer.protection privacy ``` ## Error: `FsS3UnableToConnectToEndpoint` DataSync can't connect to your [Amazon S3 location](create-s3-location.md). This could mean the location's S3 bucket isn't reachable or the location isn't configured correctly. Do the following until you resolve the issue: + Check if DataSync can [access your S3 bucket](create-s3-location.md#create-s3-location-access). + Make your sure location is configured correctly by using the DataSync console or [DescribeLocationS3](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeLocationS3.html) operation. ## Error: `FsS3HeadBucketFailed` DataSync can't access the S3 bucket that you're transferring to or from. Check if DataSync has permission to access the bucket by using the Amazon S3 [HeadBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) operation. If you need to adjust your permissions, see [Providing DataSync access to S3 buckets](create-s3-location.md#create-s3-location-access). ## Task fails with an `Unable to list Azure Blobs on the volume root` error If your DataSync transfer task fails with an `Unable to list Azure Blobs on the volume root` error, there might be an issue with your shared access signature (SAS) token or your Azure storage account's network. **Actions to take** Try the following and run your task again until you fix the issue: + Make sure that your [SAS token](creating-azure-blob-location.md#azure-blob-sas-tokens) has the right permissions to access your Microsoft Azure Blob Storage. + If you're running your DataSync agent in Azure, configure your storage account to allow access from the virtual network where your agent resides. + If you're running your agent on Amazon EC2, configure your Azure storage firewall to allow access from the agent's public IP address. For information on how to configure your Azure storage account's network, see the [Azure Blob Storage documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security). ## Error: `FsAzureBlobVolRootListBlobsFailed` The shared access signature (SAS) token that DataSync uses to access your Microsoft Azure Blob Storage doesn't have the List permission. To resolve the issue, [update your location](creating-azure-blob-location.md#azure-blob-update-location) with a token that has the List permission and try running your task again. ## Error: `SrcLocHitAccess` DataSync can't access your source location. Check that DataSync has permission to access the location and try running your task again. ## Error: `SyncTaskErrorLocationNotAdded` DataSync can't access your location. Check that DataSync has permission to access the location and try running your task again. ## Error: `S3 location creation failed with (InvalidRequestException) when calling the CreateLocationS3 operation` This error could be related to IAM permissions, Amazon S3 bucket policies, AWS KMS permissions or other permission issues. If you get this error, use the following information to troubleshoot: + [Troubleshoot access denied (403 Forbidden) errors in Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html) in the *Amazon Simple Storage Service User Guide* + [How do I troubleshoot 403 Access Denied errors from Amazon S3?](https://repost.aws/knowledge-center/s3-troubleshoot-403) on AWS re:Post ## Task with S3 source location fails with `HeadObject` or `GetObjectTagging` error **Errors related to `HeadObject` or `GetObjectTagging`** If you're transferring objects with specific version IDs from an S3 bucket, you might see an error related to `HeadObject` or `GetObjectTagging`. For example, here's an error related to `GetObjectTagging`: ``` [WARN] Failed to read metadata for file /picture1.png (versionId: 111111): S3 Get Object Tagging Failed [ERROR] S3 Exception: op=GetObjectTagging photos/picture1.png, code=403, type=15, exception=AccessDenied, msg=Access Denied req-hdrs: content-type=application/xml, x-amz-api-version=2006-03-01 rsp-hdrs: content-type=application/xml, date=Wed, 07 Feb 2024 20:16:14 GMT, server=AmazonS3, transfer-encoding=chunked, x-amz-id-2=IOWQ4fDEXAMPLEQM+ey7N9WgVhSnQ6JEXAMPLEZb7hSQDASK+Jd1vEXAMPLEa3Km, x-amz-request-id=79104EXAMPLEB723 ``` If you see either of these errors, validate that the IAM role that DataSync uses to access your S3 source location has the following permissions: + `s3:GetObjectVersion` + `s3:GetObjectVersionTagging` If you need to update your role with these permissions, see [Creating an IAM role for DataSync to access your Amazon S3 location](create-s3-location.md#create-role-manually). # Troubleshooting issues with DataSync tasks Use the following information to help you troubleshoot issues with AWS DataSync tasks and task executions. These issues might include task setup problems, stuck task executions, and data not transferring as expected. ## Error: Invalid SyncOption value. Option: TransferMode,PreserveDeletedFiles, Value: ALL,REMOVE. This error occurs when you're creating or editing your DataSync task and you select the **Transfer all data** option and deselect the **Keep deleted files** option. When you transfer all data, DataSync doesn't scan your destination location and doesn't know what to delete. ## Task execution fails with an EniNotFound error This error occurs if you delete one of your task's network interfaces in your virtual private cloud (VPC). If your task is scheduled or queued, the task will fail if it's missing a [network interface required to transfer your data](required-network-interfaces.md). **Actions to take** You have the following options to work around this issue: + Manually restart the task. When you do this, DataSync will create any missing network interfaces it needs to run the task. + If you need to clean up resources in your VPC, make sure that you don't delete network interfaces related to a DataSync task that you're still using. To see the network interfaces allocated to your task, do one of the following: + Use the [DescribeTask](https://docs.aws.amazon.com//datasync/latest/userguide/API_DescribeTask.html) operation. You can view the network interfaces in the `SourceNetworkInterfaceArns` and `DestinationNetworkInterfaceArns` response elements. + In the Amazon EC2 console, search for your task ID (such as `task-f012345678abcdef0`) to find its network interfaces. + Consider not running your tasks automatically. This could include disabling task queueing or scheduling (through DataSync or custom automation). ## Task execution fails with a Cannot allocate memory error When your DataSync task fails with a Cannot allocate memory error, it can mean a few different things. **Action to take** Try the following until you no longer see the issue: + If your transfer involves an agent, make sure that the agent meets the [virtual machine (VM)](agent-requirements.md#hardware) or [Amazon EC2 instance](agent-requirements.md#ec2-instance-types) requirements. + Split your transfer into multiple tasks by using [filters](filtering.md). It's possible that you're trying to transfer more files or objects than what [one DataSync task can handle](datasync-limits.md#task-hard-limits). + If you still see the issue, [contact Support](https://aws.amazon.com/contact-us/). ## Task fails with `Input/Output error` for FSx for ONTAP file system When your DataSync task fails with an `Input/Output error` when transferring data with an FSx for ONTAP file system, it can be due to one or more of the following issues. **The FSx for ONTAP volume has reached its maximum file capacity** This error occurs when the number of available inodes, or file pointers, on a volume is exhausted. **Actions to take** First, view the volume's [maximum file capacity](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/view-volume-file-capacity.html). Then, increase the volume's file capacity by either increasing the number of inodes or by increasing the storage capacity. For more information, see [Increasing a volume's maximum file capacity](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/low-volume-capacity.html#max-file-capacity) in the *FSx for ONTAP User Guide*. **The FSx for ONTAP volume has run out of available storage capacity** This error occurs when the volume doesn't have available storage capacity. **Actions to take** First, determine the volume's [available storage capacity](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/monitor-volume-storage-console.html). Then, increase the volume's storage capacity. For more information, see [Increasing a volume's storage capacity](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/low-volume-capacity.html#increase-volume-capacity) in the *FSx for ONTAP User Guide*. **Note** To automatically increase the volume's storage capacity when needed, see [Using volume autosizing](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/low-volume-capacity.html#volume-autosizing) in the *FSx for ONTAP User Guide*. **The FSx for ONTAP directory has reached the maximum number of files that can be stored in each directory** This error occurs when you have reached the maximum number of files that can be stored in each directory. **Action to take** Increase the max directory size to support larger directories. For more information, see [Best practices for using the FSx for ONTAP maximum directory size](https://docs.aws.amazon.com/prescriptive-guidance/latest/fsx-ontap-enterprise-deployment/best-practices.html#bp-max-directory-size) in *AWS Prescriptive Guidance*. **The DataSync task execution generated too much read write concurrency, consuming a high percentage of the file system's throughput capacity** This error occurs when the DataSync task execution is consuming too much of your file system's available throughput capacity. **Actions to take** First, determine whether the task execution was consuming too much of the file system's throughput capacity using the following methods: + Monitor the file system’s performance using the available CloudWatch metrics. For more information, see [Monitoring file system metrics](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/monitor-throughput-cloudwatch.html#fsxn-howtomonitor-fs) in the *FSx for ONTAP User Guide*. + Monitor the file system for file server performance warnings in the Amazon FSx console. For more information, see [Performance warnings and recommendations](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/performance-insights-FSxN.html#resolve-warnings) in the *FSx for ONTAP User Guide*. Then, make sure that the task doesn't use all of the file system's available throughput capacity by doing one of the following: + Set the task execution's bandwidth limit to an amount that is less than the FSx for ONTAP file system's provisioned throughput capacity. For more information, see [Setting bandwidth limits for your AWS DataSync task](configure-bandwidth.md). + Increase the file system’s provisioned throughput capacity. For more information, see [Updating throughput capacity](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/increase-throughput-capacity.html) in the *FSx for ONTAP User Guide*. ## Task fails with `Connection Reset by peer` or `Host is down` message for FSx for ONTAP file system If your DataSync task fails with a `Connection Reset by peer` or `Host is down` message when transferring data with an FSx for ONTAP file system, it can be due to one or more of the following issues: + The file system’s SMB server was rebooted or otherwise disconnected during the task execution. + The file system failed over from the primary to secondary server (and IP address) during task execution. DataSync does not support failing over to a secondary IP address during task execution. FSx for ONTAP file systems failover to a secondary server and IP address during the following events: + The primary server becomes unavailable. + The primary server's Availability Zone becomes unavailable (for Multi-AZ file systems). + During a user-initiated throughput capacity change. + During the file system's regularly scheduled maintenance window. For more information, see [FSx for ONTAP Failover process](https://docs.aws.amazon.com/fsx/latest/ONTAPGuide/high-availability-AZ.html#Failover) in the FSx for ONTAP User Guide. **Action to take** Restart the task. ## Task execution has a launching status but nothing seems to be happening Your DataSync task can get stuck with a **Launching** status typically because the agent is powered off or has lost network connectivity. **Actions to take** Make sure that your agent's status is **ONLINE**. If the agent is **OFFLINE**, make sure it's powered on. If the agent is powered on and the task is still **Launching**, then there's likely a network connection problem between your agent and AWS. For information about how to test network connectivity, see [Verifying your agent's connection to the DataSync service](test-agent-connections.md#test-network). If you're still having this issue, see [I don't know what's going on with my agent. Can someone help me?](troubleshooting-datasync-agents.md#enable-support-access). ## Task execution seems stuck in the preparing status The amount of time your DataSync transfer task has the **Preparing** status depends on the amount of data in your transfer source and destination and the performance of those storage systems. When a task starts, DataSync performs a recursive directory listing to discover all files, objects, directories, and metadata in your source and destination. DataSync uses these listings to identify differences between storage systems and determine what to copy. This process can take a few minutes or even a few hours. **Action to take** You shouldn't have to do anything. Continue to wait for the task status to change to **Transferring**. If the status still doesn't change, contact [AWS Support Center](https://console.aws.amazon.com/support/home#/). ## Task execution stops before the transfer finishes If your DataSync task execution stops early, your task configuration might include an AWS Region that's disabled in your AWS account. **Actions to take** Do the following to run your task again: 1. Check the [opt-in status](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-regions.html) of your task's Regions and make sure they're enabled. 1. [Start the task](run-task.md) again. ## Task execution fails when transferring from a Google Cloud Storage bucket Because DataSync communicates with Google Cloud Storage by using the Amazon S3 API, there's a limitation that might cause your DataSync transfer to fail if you try to copy object tags. The following message related to the issue appears in your CloudWatch logs: [WARN] Failed to read metadata for file /*your-bucket*/*your-object*: S3 Get Object Tagging Failed: proceeding without tagging To prevent this, deselect the **Copy object tags** option when configuring your transfer task settings. ## There are mismatches between task execution's timestamps When looking at the DataSync console or Amazon CloudWatch logs, you might notice that the start and end times for your DataSync task execution don't match the timestamps you see in other monitoring tools. This is because the console and CloudWatch logs take into account the time a task execution spends in the launching or queueing [states](run-task.md#understand-task-execution-statuses), while some other tools don’t. You might notice this discrepancy when comparing execution timestamps between the DataSync console or CloudWatch logs and the following places: + Logs for the file system involved in your transfer + The last modified date on an Amazon S3 object that DataSync wrote to + Network traffic coming from the DataSync agent + Amazon EventBridge events ## Task execution fails with `NoMem` error The set of data you're trying to transfer may be too large for DataSync. If you see this error, contact [AWS Support Center](https://console.aws.amazon.com/support/home#/). ## Task execution fails with `FsNfsIdMappingEnabled` error DataSync does not support NFSv4 ID mapping. To work around this, [configure your NFS location to use NFSv3](create-nfs-location.md#configure-network-nfs-location). ## Object fails to transfer to Azure Blob Storage with `user metadata key` error When transferring from an S3 bucket to Azure Blob Storage, you might see the following error: ``` [ERROR] Failed to transfer file /user-metadata/file1: Azure Blob user metadata key must be a CSharp identifier ``` This means that `/user-metadata/file1` includes user metadata that doesn't use a valid C\$1 identifier. For more information, see the [Microsoft documentation](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/identifier-names). ## There's an `/.aws-datasync` folder in the destination location DataSync creates a folder called `/.aws-datasync` in your destination location to help facilitate your data transfer. While DataSync typically deletes this folder following your transfer, there might be situations where this doesn't happen. **Action to take** Delete this folder anytime as long as you don't have a running task execution copying to that location. ## Can't transfer symbolic links between locations using SMB When your task execution finishes, you see the following error: ``` Transfer and verification completed. Selected files transferred except for files skipped due to errors. If no skipped files are listed in Cloud Watch Logs, please contact AWS Support for further assistance. ``` When transferring between SMB storage systems (such as an SMB file server and Amazon FSx for Windows File Server file system), you might see the following warnings and errors in your CloudWatch logs: ``` [WARN] Failed to read metadata for file /appraiser/symlink: No data available [ERROR] Failed to read metadata for directory /appraiser/symlink: No data available ``` **Action to take** DataSync doesn't support transferring symbolic links (or hard links) when transferring between these location types. For more information, see [Links and directories copied by AWS DataSync](special-files-copied.md). ## Task report errors You might run into one of the following errors while trying to monitor your DataSync transfer with a task report. | Error message | Workaround | | --- | --- | | File path exceeds the maximum length of 4,096 characters. Cannot write to Task Report | None. DataSync can't transfer a file with a path that exceeds 4,096 bytes. For more information, see [Storage system, file, and object limits](datasync-limits.md#file-system-limits). | | Failed to upload Task Report(s) to S3 due to an invalid bucket or IAM role | Check that the [DataSync IAM role](creating-task-report.md#task-report-access) has the right permissions to upload a task report to your S3 bucket. | | Execution error occurred prior to generating any Task Reports | Check your [CloudWatch logs](monitor-datasync.md) to identify why your task execution failed. | # Troubleshooting data verification issues By default, AWS DataSync [verifies the integrity](how-datasync-transfer-works.md#how-verifying-works) of your data at the end of a transfer. Use the following information to help you diagnose common verification errors and warnings, such as files being modified or deleted before DataSync finishes verifying your data. With verification issues, many times it helps to review your [CloudWatch logs](configure-logging.md) (or [task reports](task-reports.md)) in addition to the task execution error that you're seeing. DataSync provides JSON-structured logs for Enhanced mode tasks, while Basic mode tasks have unstructured logs. ## There are mismatches between a file's contents When your task execution finishes, you see the following error: ``` Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs ``` In your CloudWatch logs, you might notice failed verifications for contents that differ between the source and destination locations. This can happen if files are modified during your transfer. For example, the following logs shows that `file1.txt` has different `mtime`, `srcHash`, and `dstHash` values: **Basic mode log example** ``` [NOTICE] Verification failed <> /directory1/directory2/file1.txt [NOTICE] /directory1/directory2/file1.txt srcMeta: type=R mode=0755 uid=65534 gid=65534 size=534528 atime=1633100003/684349800 mtime=1602647222/222919600 extAttrsHash=0 [NOTICE] srcHash: 0c506c26bd1e43bd3ac346734f1a9c16c4ad100d1b43c2903772ca894fd24e44 [NOTICE] /directory1/directory2/file1.txt dstMeta: type=R mode=0755 uid=65534 gid=65534 size=511001 atime=1633100003/684349800 mtime=1633106855/859227500 extAttrsHash=0 [NOTICE] dstHash: dbd798929f11a7c0201e97f7a61191a83b4e010a449dfc79fbb8233801067c46 ``` In DataSync, `mtime` represents the last time a file was written to before [preparation](how-datasync-transfer-works.md#how-datasync-prepares). When verifying transfers, DataSync compares `mtime` values between source and destination locations. A verification failure like this occurs if the `mtime` for a file isn't the same for both locations. The differences between `srcHash` and `dstHash` indicate the file's contents don't match at both locations. **Actions to take** Do the following: 1. Use an epoch time converter to determine whether the source or destination file or object was modified more recently. This can help identify which version is current. 1. To avoid this error again, [schedule your task](task-scheduling.md) to run during a maintenance window when there's no activity at your source and destination. ## There's a mismatch between a file's SMB metadata When your task execution finishes, you see the following error: ``` Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs ``` When transferring between storage systems that support the Server Message Block (SMB) protocol, you might see this error when a file's extended SMB attributes don't match between source and destination. For example, the following logs show that `file1.txt` has a different `extAttrsHash` value between locations, indicating the file contents are identical but extended attributes weren't set at the destination: **Basic mode log example** ``` [NOTICE] Verification failed <> /directory1/directory2/file1.txt [NOTICE] /directory1/directory2/file1.txt srcMeta: type=R mode=0755 uid=65534 gid=65534 size=1469752 atime=1631354985/174924200 mtime=1536995541/986211400 extAttrsHash=2272191894 [NOTICE] srcHash: 38571d42b646ac8f4034b7518636b37dd0899c6fc03cdaa8369be6e81a1a2bb5 [NOTICE] /directory1/directory2/file1.txt dstMeta: type=R mode=0755 uid=65534 gid=65534 size=1469752 atime=1631354985/174924200 mtime=1536995541/986211400 extAttrsHash=3051150340 [NOTICE] dstHash: 38571d42b646ac8f4034b7518636b37dd0899c6fc03cdaa8369be6e81a1a2bb5 ``` You might also see a related error message about extended attributes: ``` [ERROR] Deferred error: WriteFileExtAttr2 failed to setextattrlist(filename="/directory1/directory2/file1.txt"): Input/output error ``` **Action to take** This error typically occurs when there are insufficient permissions to copy access control lists (ACLs) to the destination. To resolve this issue, review the following configuration guides based on your destination type: + [Required permissions](create-fsx-location.md#create-fsx-windows-location-permissions) with FSx for Windows File Server file systems + [Required permissions](create-ontap-location.md#create-ontap-location-smb) with FSx for ONTAP file systems that use SMB ## Files to transfer are no longer at source location When your task execution finishes, you see the following error: ``` Transfer and verification completed. Selected files transferred except for files skipped due to errors. If no skipped files are listed in Cloud Watch Logs, please contact AWS Support for further assistance. ``` In your logs, you might see errors indicating that files aren't at the source location. This can happen if files (such as `file1.dll` and `file2.dll`) are deleted after [preparation](how-datasync-transfer-works.md#how-datasync-prepares) but before DataSync transfers them: **Basic mode log example** ``` [ERROR] Failed to open source file /file1.dll: No such file or directory [ERROR] Failed to open source file /file2.dll: No such file or directory ``` **Action to take** To avoid these situations, [schedule your task](task-scheduling.md) to run when there's no activity at the source location. For example, you can run your task during a maintenance window when users and applications aren't actively working with that location. In some cases, you might not see logs associated with this error. If that happens, contact [AWS Support Center](https://console.aws.amazon.com/support/home#/). ## DataSync can't verify destination data When your task execution finishes, you see the following error: ``` Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs ``` In your logs, you might notice that DataSync can't verify certain folders or files in the destination location. These errors can look like this: **Basic mode log example** ``` [ERROR] Failed to read metadata for destination file /directory1/directory2/file1.txt: No such file or directory ``` For files, you might see verification failures like this: **Basic mode log example** ``` [NOTICE] Verification failed <> /directory1/directory2/file1.txt [NOTICE] /directory1/directory2/file1.txt srcMeta: type=R mode=0755 uid=65534 gid=65534 size=61533 atime=1633099987/747713800 mtime=1536995631/894267700 extAttrsHash=232104771 [NOTICE] srcHash: 1426fe40f669a7d36cca1b5329983df31a9aeff8eb9fe3ac885f26de2f8fff6b [NOTICE] /directory1/directory2/file1.txt dstMeta: type=R mode=0755 uid=65534 gid=65534 size=0 atime=0/0 mtime=0/0 extAttrsHash=0 [NOTICE] dstHash: 0000000000000000000000000000000000000000000000000000000000000000 ``` **Action to take** These logs indicate that destination data was deleted after the transfer but before verification. (Logs look similar when data is uploaded to a source location during the same time frame.) To avoid these situations, [schedule your task](task-scheduling.md) to run when there's no activity at the destination location. For example, you can run your task during a maintenance window when users and applications aren't actively working with that location. ## DataSync can't read object metadata When your task execution finishes, you see the following error: ``` Transfer and verification completed. Selected files transferred except for files skipped due to errors. If no skipped files are listed in Cloud Watch Logs, please contact AWS Support for further assistance. ``` In your logs, you might notice that DataSync can't read `file1.png` because of a failed Amazon S3 `HeadObject` request. [DataSync makes `HeadObject` requests](create-s3-location.md#create-s3-location-s3-requests-made) with S3 locations during task preparation and verification. **Basic mode log example** ``` [WARN] Failed to read metadata for file /file1.png: S3 Head Object Failed ``` **Actions to take** To fix this issue, verify whether DataSync has the right level of permissions to work with your S3 bucket: + Make sure that the IAM role that DataSync uses to access your Amazon S3 locations allows the `s3:GetObject` permission. For more information, see [Required permissions](create-s3-location.md#create-s3-location-required-permissions). + If your S3 bucket uses server-side encryption, make sure that DataSync is allowed to access the objects in that bucket. For more information, see [Accessing S3 buckets using server-side encryption](create-s3-location.md#create-s3-location-encryption). ## There's a mismatch in an object's system-defined metadata When your Enhanced mode task execution between S3 buckets finishes, you see the following error: ``` Verification failed due to a difference in metadata ``` You might notice in your logs a mismatch in an object’s Amazon S3 [system-defined metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#SysMetadata). In this particular example, the source object doesn't have `Content-Type` metadata but the destination object does. This happened because the destination S3 bucket automatically applied `"ContentType": "application/octet-stream"` metadata to the object when DataSync transferred it there. **Enhanced mode log example** ``` { "Action": "VERIFY", "Source": { "LocationId": "loc-0b3017fc4ba4a2d8d", "RelativePath": "encoding/content-null", "Metadata": { "Type": "Object", "ContentSize": 24, "LastModified": "2024-12-23T15:48:15Z", "S3": { "SystemMetadata": { "ETag": "\"68b9c323bb846841ee491481f576ed4a\"" }, "UserMetadata": {}, "Tags": {} } } }, "Destination": { "LocationId": "loc-abcdef01234567890", "RelativePath": "encoding/content-null", "Metadata": { "Type": "Object", "ContentSize": 24, "LastModified": "2024-12-23T16:00:03Z", "S3": { "SystemMetadata": { "ContentType": "application/octet-stream", "ETag": "\"68b9c323bb846841ee491481f576ed4a\"" }, "UserMetadata": { "file-mtime": "1734968895000" }, "Tags": {} } } }, "TransferType": "CONTENT_AND_METADATA", "ErrorCode": "MetadataDiffers", "ErrorDetail": "Verification failed due to a difference in metadata" } ``` **Action to take** To avoid this error, update your source location objects to include the `Content-Type` metadata property. ## Understanding data verification duration DataSync verification includes an SHA256 checksum on file content and an exact comparison of file metadata between locations. How long verification takes depends on several factors, including the number of files or objects involved, the size of the data in the storage systems, and the performance of these systems. **Action to take** Given the factors that can affect verification time, you shouldn't have to do anything. However, if your task execution seems stuck with a [verifying](run-task.md#understand-task-execution-statuses) status, contact [AWS Support Center](https://console.aws.amazon.com/support/home#/). # Troubleshooting higher than expected S3 storage costs with DataSync If your Amazon S3 storage costs are higher than you thought they would be following an AWS DataSync transfer, it might be due to one or more of the following reasons: + When transferring to or from S3 buckets, you incur costs related to S3 API requests made by DataSync. + DataSync uses the Amazon S3 multipart upload feature to upload objects to S3 buckets. This approach can result in unexpected storage charges for uploads that don't complete successfully. + DataSync copies object tags from source and destination objects when **Copy object tags** is enabled on the console or `ObjectTags` is set to `PRESERVE`. Copying of these object tags can incur S3 API request costs. + Object versioning might be enabled on your S3 bucket. Object versioning results in Amazon S3 storing multiple copies of objects that have the same name. **Actions to take** In these cases, you can take the following steps: + Make sure you understand how DataSync uses S3 requests and how they might be affecting your storage costs. For more information, see [Evaluating S3 request costs when using DataSync](create-s3-location.md#create-s3-location-s3-requests). + If the issue is related to multipart uploads, configure a policy for multipart uploads for your S3 bucket to clean up incomplete multipart uploads to reduce storage cost. For more information, see the AWS blog post [S3 Lifecycle Management Update - Support for Multipart Uploads and Delete Markers](https://aws.amazon.com/blogs/aws/s3-lifecycle-management-update-support-for-multipart-uploads-and-delete-markers/). + If the issue is related to copying object tags and you don't need object tags, clear the **Copy object tags** checkbox in the DataSync console or set `ObjectTags` to `None` when creating, starting, or updating a task. + If the issue is related to object versioning, disable object versioning on your S3 bucket. f you need additional help, contact [AWS Support Center](https://console.aws.amazon.com/support/home#/).