# Monitoring your AWS DataSync transfers
Monitoring is important for maintaining the reliability and performance of your AWS DataSync transfer activities. We recommend that you collect monitoring data so that you can more easily debug errors if one occurs. Before you start monitoring DataSync, however, create a monitoring plan that includes answers to the following questions:
+ What are your monitoring goals?
+ What resources will you monitor?
+ How often will you monitor these resources?
+ What monitoring tools will you use?
+ Who will perform the monitoring tasks?
+ Who should be notified when something goes wrong?
AWS provides various services and tools for monitoring DataSync. You can configure some of these to do the monitoring for you, but some require manual intervention. We recommend that you automate monitoring tasks as much as possible.
**Topics**
+ [Understanding data transfer performance counters](transfer-performance-counters.md)
+ [Monitoring data transfers with Amazon CloudWatch metrics](monitor-datasync.md)
+ [Monitoring your data transfers with task reports](task-reports.md)
+ [Monitoring data transfers with Amazon CloudWatch Logs](configure-logging.md)
+ [Logging AWS DataSync API calls with AWS CloudTrail](logging-using-cloudtrail.md)
+ [Monitoring events by using Amazon EventBridge](events.md)
+ [Monitoring AWS DataSync with manual tools](monitoring-task-manually.md)
# Understanding data transfer performance counters
When you [start a task](run-task.md), AWS DataSync provides counters to help track your data transfer's performance and progress.
Use the following information to understand what each counter represents. You can view these counters in the DataSync console or a [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html) response. Some counters aren't available with every [task mode](choosing-task-mode.md).
| Console | `DescribeTaskExecution` | Task mode support | Description |
| --- | --- | --- | --- |
| – | `BytesWritten` | Enhanced, Basic | The number of logical bytes that DataSync actually writes to the destination location. |
| **Data throughput** | – | Enhanced, Basic | The rate at which DataSync writes logical bytes to the destination location. If you're using [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html), how you calculate this counter depends on your task mode: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| Data transferred | `BytesTransferred` | Enhanced, Basic | The number of bytes that DataSync sends to the network before compression (if compression is possible). For the number of bytes transferred over the network, see the **Network throughput** (in console) or `BytesCompressed` (in [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html)) counter. |
| **Deleted from destination** | `FilesDeleted` | Basic | The number of files, objects, and directories that DataSync actually deletes in your destination location. If you don't configure your task to [delete data in the destination that isn't in the source](configure-metadata.md#task-option-file-object-handling): [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| **Deleted from destination** | `FilesDeleted`, `FoldersDeleted` | Enhanced | The number of files or objects, and directories that DataSync actually deletes in your destination location. If you don't configure your task to [delete data in the destination that isn't in the source](configure-metadata.md#task-option-file-object-handling): [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| – | `EstimatedBytesToTransfer` | Enhanced, Basic | The number of logical bytes that DataSync expects to write to the destination location. |
| – | `EstimatedFilesToDelete` | Basic | The number of files, objects, and directories that DataSync expects to delete in your destination location. If you don't configure your task to [delete data in the destination that isn't in the source](configure-metadata.md#task-option-file-object-handling), the value is always `0`. |
| – | `EstimatedFilesToDelete`, `EstimatedFoldersToDelete` | Enhanced | The number of files or objects, and directories that DataSync expects to delete in your destination location. If you don't configure your task to [delete data in the destination that isn't in the source](configure-metadata.md#task-option-file-object-handling), the value is always `0`. |
| – | `EstimatedFilesToTransfer` | Basic | The number of files, objects, and directories that DataSync expects to transfer over the network. This value is calculated while DataSync [prepares](run-task.md#understand-task-execution-statuses) the transfer. How this gets calculated depends primarily on the [transfer mode](configure-metadata.md#task-option-transfer-mode) you're using: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| – | `EstimatedFilesToTransfer`, `EstimatedFoldersToTransfer` | Enhanced | The number of files or objects, and directories that DataSync expects to transfer over the network. This value is calculated while DataSync [prepares](run-task.md#understand-task-execution-statuses) the transfer. How this gets calculated depends primarily on the [transfer mode](configure-metadata.md#task-option-transfer-mode) you're using: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| **File throughput** | – | Enhanced, Basic | The rate at which DataSync transfers files, objects, and directories over the network. If you're using [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html), how you calculate this counter depends on your task mode: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| – | `FilesFailed`, `FoldersFailed` | Enhanced | The number of files or objects, and directories that DataSync fails to prepare, transfer, verify, and delete during your task execution. If there are failures, you can see these alongside the **Prepared**, **Transferred**, **Skipped**, and **Deleted from destination** console counters, respectively. |
| **Listed at source** | `FilesListed.AtSource`, `FoldersListed.AtSource` | Enhanced | The number of files or objects, and directories that DataSync finds at your source location. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| – | `FilesListed.AtDestinationForDelete`, `FoldersListed.AtDestinationForDelete` | Enhanced | The number of files or objects, and directories that DataSync finds at your destination location. This counter is only applicable if you configure your task to [delete data in the destination that isn't in the source](configure-metadata.md#task-option-file-object-handling). |
| **Network throughput**\$1 | `BytesCompressed` | Enhanced, Basic | The number of physical bytes that DataSync transfers over the network after compression (if compression is possible). This number is typically less than **Data transferred** (in console) or `BytesTransferred` (in [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html)) unless the data isn't compressible. \$1 For Enhanced mode, **Network throughput** doesn't display in the console. |
| **Percent compressed** | – | Enhanced, Basic | The percentage of transfer data that DataSync compressed before sending it over the network. If you're using [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html), you can calculate this counter with `1 - BytesCompressed / BytesWritten`. |
| **Prepared** | `FilesPrepared`, `FoldersPrepared` | Enhanced | The number of files or objects, and directories that DataSync will attempt to transfer after comparing your source and destination locations. In the console, this counter can also show you the number of objects that DataSync skips during preparation. For more information, see [How DataSync prepares your data transfer](how-datasync-transfer-works.md#how-datasync-prepares). This counter isn't applicable if you configure your task to [transfer all data](configure-metadata.md#task-option-transfer-mode). In that scenario, DataSync copies everything from the source to the destination without comparing differences between locations. |
| **Processing rate** | – | Enhanced, Basic | The rate at which DataSync reads files, objects, and directories at your source location. The processing rate is based on several [CloudWatch metrics](monitor-datasync.md#accessing-metrics). The exact metrics depend on the task mode you're using. **Enhanced mode**: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) **Basic mode:** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/datasync/latest/userguide/transfer-performance-counters.html) |
| **Remaining** | – | Basic | The remaining number of files, objects, and directories that DataSync expects to transfer over the network. If you're using [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html), you can calculate this counter by subtracting `FilesTransferred` from `EstimatedFilesToTransfer`. |
| **Skipped**\$1 | `FilesSkipped` | Basic | The number of files, objects, and directories that DataSync skips during your transfer. |
| – | `FilesSkipped`, `FoldersSkipped` | Enhanced | The number of files or objects, and directories that DataSync skips during your transfer. **Skipped** items are included in the **Prepared** counter when [transferring only the data that has changed](configure-metadata.md#task-option-transfer-mode) or the **Transferred** counter when [transferring all data](configure-metadata.md#task-option-transfer-mode). |
| **Transferred** | `FilesTransferred` | Basic | The number of files, objects, and directories that DataSync transfers over the network. This value is updated periodically during your task execution when something is read from the source and sent over the network. If DataSync fails to transfer something, this value can be less than `EstimatedFilesToTransfer` or `EstimatedFoldersToTransfer`. In some cases, this value can also be greater than `EstimatedFilesToTransfer` or `EstimatedFoldersToTransfer`. This counter is implementation-specific for some location types, so don't use it as an exact indication of what's transferring or to monitor your task execution. |
| **Transferred** | `FilesTransferred`, `FoldersTransferred` | Enhanced | The number of files or objects, and directories that DataSync transfers over the network. This value is updated periodically during your task execution when something is read from the source and sent over the network. If DataSync fails to transfer something, this value can be less than `EstimatedFilesToTransfer` or `EstimatedFoldersToTransfer`. In some cases, this value can also be greater than `EstimatedFilesToTransfer` or `EstimatedFoldersToTransfer`. This counter is implementation-specific for some location types, so don't use it as an exact indication of what's transferring or to monitor your task execution. |
| **Verified** | `FilesVerified` | Basic | The number of files, objects, and directories that DataSync verifies during your transfer. When you configure your task to [verify only transferred data](configure-data-verification-options.md), DataSync doesn't verify directories in some situations or files or objects that fail to transfer. |
| **Verified** | `FilesVerified`, `FoldersVerified` | Enhanced | The number of files or objects, and directories that DataSync verifies during your transfer. |
# Monitoring data transfers with Amazon CloudWatch metrics
Amazon CloudWatch provides metrics to track DataSync transfer performance and troubleshoot issues with your transfer task.
You can monitor AWS DataSync transfer performance by using Amazon CloudWatch metrics. DataSync metrics are automatically sent to CloudWatch in 5-minute intervals (regardless of how you [configure logging](configure-logging.md)). The metrics are retained for a period of 15 months.
To see CloudWatch metrics for DataSync, you can use the following tools:
+ The CloudWatch console
+ The CloudWatch CLI
+ The CloudWatch API
+ The DataSync console (on the task execution's details page)
For more information, see the [https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html).
## CloudWatch metrics for DataSync
DataSync metrics use the `aws/datasync` namespace and provide metrics for the following dimensions:
+ **AgentId** – The unique ID of the agent (if your task uses an agent).
+ **TaskId** – The unique ID of the task. It takes the form of `task-01234567890abcdef`.
The `aws/datasync` namespace includes the following metrics. Some metrics aren't available with every [task mode](choosing-task-mode.md).
| CloudWatch metric | Task mode support | Description |
| --- | --- | --- |
| `BytesCompressed` | Basic | The number of physical bytes that DataSync transfers over the network after compression (if compression is possible). This number is typically less than `BytesTransferred` unless the data isn't compressible. Unit: Bytes |
| `BytesPreparedDestination` | Basic | The number of logical bytes that DataSync prepares at the destination location. Unit: Bytes |
| `BytesPreparedSource` | Basic | The number of logical bytes that DataSync prepares at the source location. Unit: Bytes |
| `BytesTransferred` | Basic | The number of bytes that DataSync sends to the network before compression (if compression is possible). For the number of bytes transferred over the network, see the `BytesCompressed` metric. Unit: Bytes |
| `BytesVerifiedDestination` | Basic | The number of logical bytes that DataSync verifies at the destination location. Unit: Bytes |
| `BytesVerifiedSource` | Basic | The number of logical bytes that DataSync verifies at the source location. Units: Bytes |
| `BytesWritten` | Enhanced, Basic | The number of logical bytes that DataSync writes to the destination location. Unit: Bytes |
| `FilesDeleted` | Enhanced, Basic | The number of files, objects, and directories that DataSync deletes in your destination location. If you don't configure your task to [delete data in the destination that isn't in the source](configure-metadata.md#task-option-file-object-handling), the value is always `0`. Unit: Count |
| `FilesListedSource` | Enhanced | The number of objects that DataSync finds at your source location. Unit: Count |
| `FilesPrepared` | Enhanced | The number of objects that DataSync will attempt to transfer after comparing your source and destination locations. For more information, see [How DataSync prepares your data transfer](how-datasync-transfer-works.md#how-datasync-prepares). This metric isn't applicable if you configure your task to [transfer all data](configure-metadata.md#task-option-transfer-mode). In that scenario, DataSync copies everything from the source to the destination without comparing differences between the locations. Unit: Count |
| `FilesPreparedDestination` | Basic | The number of files, objects, and directories that DataSync prepares at the destination location. Unit: Count |
| `FilesPreparedSource` | Basic | The number of files, objects, and directories that DataSync prepares at the source location. Unit: Count |
| `FilesSkipped` | Basic | The number of files, objects, and directories that DataSync skips during your transfer. Unit: Count |
| `FilesTransferred` | Enhanced, Basic | The number of files, objects, and directories that DataSync transfers over the network. This value is updated periodically during the [task execution](run-task.md#understand-task-execution-statuses) when something is read from the source and sent over the network. This value can be less than `EstimatedFilesToTransfer` in a [DescribeTaskExecution](https://docs.aws.amazon.com/datasync/latest/userguide/API_DescribeTaskExecution.html) response if DataSync fails to transfer something. In some cases, this value can also be greater than `EstimatedFilesToTransfer`. This metric is implementation-specific for some location types, so don't use it as an exact indication of what transferred or to monitor your task execution. Unit: Count |
| `FilesVerified` | Enhanced | The number of objects that DataSync verifies during your transfer. Unit: Count |
| `FilesVerifiedDestination` | Basic | The number of files, objects, and directories that DataSync verifies at the destination location. Unit: Count |
| `FilesVerifiedSource` | Basic | The number of files, objects, and directories that DataSync verifies at the source location. Unit: Count |
# Monitoring your data transfers with task reports
*Task reports* provide detailed information about what AWS DataSync attempts to transfer, skip, verify, and delete during a task execution. For more information, see [How DataSync transfers files, objects, and directories](how-datasync-transfer-works.md#transferring-files).
Task reports are generated in JSON format. You can customize the level of detail in your reports:
+ [Summary only task reports](#task-report-types-summary) give you the necessary details about your task execution, such as how many files transferred and whether DataSync could verify the data integrity of those files.
+ [Standard task reports](#task-report-types-standard) include a summary plus detailed reports that list each file, object, or folder that DataSync attempts to transfer, skip, verify, and delete. With a standard task report, you can also specify the [report level](#task-report-level) to show only the task execution's errors or its successes and errors.
## Use cases
Here are some situations where task reports can help you monitor and audit your data transfers:
+ When migrating millions of files, quickly identify files that DataSync has issues transferring.
+ Verify chain-of-custody processes for your files.
## Summary only task reports
A report that's only a summary of a task execution includes the following details:
+ The AWS account that ran the task execution
+ The source and destination locations
+ The total number of files, objects, and folders that were skipped, transferred, verified, and deleted
+ The total bytes (logical and physical) that were transferred
+ If the task execution was completed, canceled, or encountered an error
+ The start and end times (including the total time of the transfer)
+ The task's settings (such as bandwidth limits, data integrity verification, and other options for your DataSync transfer)
## Standard task reports
A standard task report includes a [summary](#task-report-types-summary) of your task execution plus detailed reports of what DataSync attempts to transfer, skip, verify, and delete.
**Topics**
+ [Report level](#task-report-level)
+ [Transferred reports](#task-report-types-transferred)
+ [Skipped reports](#task-report-types-skipped)
+ [Verified reports](#task-report-types-verified)
+ [Deleted reports](#task-report-types-deleted)
### Report level
With standard task reports, you can choose one of the following report levels:
+ Errors only
+ Successes and errors (essentially a list of everything that happened during your task execution)
For example, you might want to see which files DataSync skipped successfully during your transfer and which ones it didn't. Files that DataSync skipped successfully might be ones that you purposely want DataSync to exclude because they already exist in your destination location. However, a skipped error for instance might indicate that DataSync doesn't have the right permissions to read a file.
### Transferred reports
A list of files, objects, and directories that DataSync attempted to transfer during your task execution. A transferred report includes the following details:
+ The paths for the transferred data
+ What was transferred (content, metadata, or both)
+ The metadata, which includes the data type, content size (objects and files only), and more
+ The time when an item was transferred
+ The object version (if the destination is an Amazon S3 bucket that has versioning enabled)
+ If something was overwritten in the destination
+ Whether an item transferred successfully
**Note**
When moving data between S3 buckets, the prefix that you specify in your [source location](create-s3-location.md) can show up in your report (or in Amazon CloudWatch logs), even if that prefix doesn't exist as an object in your destination location. (In the DataSync console, you might also notice this prefix showing up as skipped or verified data.)
### Skipped reports
A list of files, objects, and directories that DataSync finds in your source location but didn't attempt to transfer. The reasons DataSync skips data can depend on several factors, such as how you configure your task and storage system permissions. Here are some examples:
+ There's a file that exists in your source and destination locations. The file in the source hasn't been modified since the previous task execution. Since you're [only transferring data that has changed](configure-metadata.md#task-option-transfer-mode), DataSync doesn't transfer that file next time you run your task.
+ An object that exists in both of your locations changes in your source. When you run your task, DataSync skips this object in your destination because your task doesn't [overwrite data in the destination](configure-metadata.md#task-option-file-object-handling).
+ DataSync skips an object in your source that's using an [archival storage class](create-s3-location.md#using-storage-classes) and isn't restored. You must restore an archived object for DataSync to read it.
+ DataSync skips a file, object, or directory in your source location because it can't read it. If this happens and isn't expected, check your storage's access permissions and make sure that DataSync can read what was skipped.
A skipped report includes the following details:
+ The paths for skipped data
+ The time when an item was skipped
+ The reason it was skipped
+ Whether an item was skipped successfully
**Note**
Skipped reports can be large when they include successes and errors, you configure your task to [transfer only the data that has changed](configure-metadata.md), and source data already exists in the destination.
### Verified reports
A list of files, objects, and directories that DataSync attempted to verify the integrity of during your task execution. A verified data report includes the following details:
+ The paths for verified data
+ The time when an item was verified
+ The reason for the verification error (if any)
+ The source and destination SHA256 checksums (files only)
+ Whether an item was successfully verified
Note the following about verified reports:
+ When you configure your task to [verify only transferred data](configure-data-verification-options.md), DataSync doesn't verify directories in some situations or files or objects that fail to transfer. In either case, DataSync doesn't include unverified data in this report.
+ If you're using [Enhanced mode](choosing-task-mode.md), verification might take longer than usual if you're transferring large objects.
### Deleted reports
A list of files, directories, and objects that were deleted during your task execution. DataSync generates this report only if you [configure your task](configure-metadata.md) to delete data in the destination location that isn't in the source. A deleted data report includes the following details:
+ The paths for deleted data
+ Whether an item was successfully deleted
+ The time when an item was deleted
## Example task reports
The level of detail in your task report is up to you. Here are some example transferred data reports with the following configuration:
+ **Report type** – Standard
+ **Report level** – Successes and errors
**Note**
Reports use the ISO-8601 standard for the timestamp format. Times are in UTC and measured in nanoseconds. This behavior differs from how some other task report metrics are measured. For example, [task execution details](https://docs.aws.amazon.com/datasync/latest/userguide/API_TaskExecutionResultDetail.html), such as `TransferDuration` and `VerifyDuration`, are measured in milliseconds.
Enhanced mode task reports use a somewhat different schema than Basic mode task reports. The following examples can help you know what to expect from your reports depending on the [task mode](choosing-task-mode.md) you use.
**Example transferred data reports with success status**
The following reports show successful transfers for an object named `object1.txt`.
```
{
"TaskExecutionId": "exec-abcdefgh12345678",
"Transferred": [{
"RelativePath": "object1.txt",
"SourceMetadata": {
"Type": "Object",
"ContentSize": 6,
"LastModified": "2024-10-04T14:40:55Z",
"SystemMetadata": {
"ContentType": "binary/octet-stream",
"ETag": "\"9b2d7e1f8054c3a2041905d0378e6f14\"",
"ServerSideEncryption": "AES256"
},
"UserMetadata": {},
"Tags": []
},
"Overwrite": "False",
"DstS3VersionId": "jtqRtX3jN4J2G8k0sFSGYK1f35KqpAVP",
"TransferTimestamp": "2024-10-04T14:48:39.748862183Z",
"TransferType": "CONTENT_AND_METADATA",
"TransferStatus": "SUCCESS"
}]
}
```
```
{
"TaskExecutionId": "exec-abcdefgh12345678",
"Transferred": [{
"RelativePath": "/object1.txt",
"SrcMetadata": {
"Type": "Regular",
"ContentSize": 6,
"Mtime": "2022-01-07T16:59:26.136114671Z",
"Atime": "2022-01-07T16:59:26.136114671Z",
"Uid": 0,
"Gid": 0,
"Mode": "0644"
},
"Overwrite": "False",
"DstS3VersionId": "jtqRtX3jN4J2G8k0sFSGYK1f35KqpAVP",
"TransferTimestamp": "2022-01-07T16:59:45.747270957Z",
"TransferType": "CONTENT_AND_METADATA",
"TransferStatus": "SUCCESS"
}]
}
```
**Example transferred data reports with error status**
The following reports provide examples of when DataSync can't transfer an object named `object1.txt`.
This report shows that DataSync can't access an object named `object1.txt` because of an AWS KMS permissions issue. (If you get an error like this, see [Accessing S3 buckets using server-side encryption](create-s3-location.md#create-s3-location-encryption).)
```
{
"TaskExecutionId": "exec-abcdefgh12345678",
"Transferred": [{
"RelativePath": "object1.txt",
"SourceMetadata": {
"Type": "Object",
"ContentSize": 6,
"LastModified": "2022-10-07T20:48:32Z",
"SystemMetadata": {
"ContentType": "binary/octet-stream",
"ETag": "\"3a7c0b2f1d9e5c4a6f8b2e0d1c9f7a3b2\"",
"ServerSideEncryption": "AES256"
},
"UserMetadata": {},
"Tags": []
},
"Overwrite": "False",
"TransferTimestamp": "2022-10-09T16:05:11.134040717Z",
"TransferType": "CONTENT_AND_METADATA",
"TransferStatus": "FAILED",
"ErrorCode": "AccessDenied",
"ErrorDetail": "User: arn:aws:sts::111222333444:assumed-role/AWSDataSyncS3Bucket/AwsSync-loc-0b3017fc4ba4a2d8d is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:kms:us-east-1:111222333444:key/1111aaaa-22bb-33cc-44d-5555eeee6666 because no identity-based policy allows the kms:GenerateDataKey action"
}]
}
```
This report shows that an object named `object1.txt` didn't transfer because of an S3 bucket permissions issue. (If you get an error like this, see [Providing DataSync access to S3 buckets](create-s3-location.md#create-s3-location-access).)
```
{
"TaskExecutionId": "exec-abcdefgh12345678",
"Transferred": [{
"RelativePath": "/object1.txt",
"SrcMetadata": {
"Type": "Regular",
"ContentSize": 6,
"Mtime": "2022-01-07T16:59:26.136114671Z",
"Atime": "2022-01-07T16:59:26.136114671Z",
"Uid": 0,
"Gid": 0,
"Mode": "0644"
},
"Overwrite": "False",
"DstS3VersionId": "jtqRtX3jN4J2G8k0sFSGYK1f35KqpAVP",
"TransferTimestamp": "2022-01-07T16:59:45.747270957Z",
"TransferType": "CONTENT_AND_METADATA",
"TransferStatus": "FAILED",
"FailureReason": "S3 Get Object Failed",
"FailureCode": 40974
}]
}
```
## Limitations
+ Individual task reports can't exceed 5 MB. If you're copying a large number of files, your task report might be split into multiple reports.
+ There are situations when creating task reports can affect the performance of your data transfer. For example, you might notice this when your network connection has high latency and the files you're transferring are small or you're copying only metadata changes.
# Creating your DataSync task reports
AWS DataSync task reports can be only a summary of your task execution or a set of detailed reports about what DataSync attempts to transfer, skip, verify, and delete.
## Prerequisites
Before you can create a task report, you must do the following.
**Topics**
+ [Create an S3 bucket for your task reports](#setting-up-task-report-create-bucket)
+ [Allow DataSync to upload task reports to your S3 bucket](#task-report-access)
### Create an S3 bucket for your task reports
If you don't already have one, [create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) where DataSync can upload your task report. Reports are stored in the S3 Standard storage class.
We recommend the following for this bucket:
+ If you're planning to transfer data to an S3 bucket, don't use the same bucket for your task report if you [disable the **Keep deleted files** option](configure-metadata.md). Otherwise, DataSync will delete any previous task reports each time you execute a task since those reports don't exist in your source location.
+ To avoid a complex access permissions setup, make sure that your task report bucket is in the same AWS account and Region as your DataSync transfer task.
### Allow DataSync to upload task reports to your S3 bucket
You must configure an AWS Identity and Access Management (IAM) role that allows DataSync to upload a task report to your S3 bucket.
In the DataSync console, you can create an IAM role that in most cases automatically includes the permissions to upload a task report to your bucket. Keep in mind that this automatically generated role might not meet your needs from a least-privilege standpoint. This role also won't work if your bucket is encrypted with a customer managed AWS Key Management Service (AWS KMS) key (SSE-KMS). In these cases, you can create the role manually as long as the role does at least the following:
+ [Prevents the cross-service confused deputy problem](https://docs.aws.amazon.com/datasync/latest/userguide/cross-service-confused-deputy-prevention.html) in the role's trusted entity.
The following full example shows how you can use the `aws:SourceArn` and `aws:SourceAccount` global condition context keys to prevent the confused deputy problem with DataSync.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "datasync.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789012"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:datasync:us-east-1:123456789012:*"
}
}
}
]
}
```
------
+ Allows DataSync to upload a task report to your S3 bucket.
The following example does this by including the `s3:PutObject` action only for a specific prefix (`reports/`) in your bucket.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [{
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::your-task-reports-bucket/reports/*"
}]
}
```
------
+ If your S3 bucket is encrypted with a customer managed SSE-KMS key, the [key's policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying.html) must include the IAM role that DataSync uses to access the bucket.
For more information, see [Accessing S3 buckets using server-side encryption](create-s3-location.md#create-s3-location-encryption).
## Creating a summary only task report
You can configure a task report that includes a [summary only](task-reports.md#task-report-types-summary) when creating your DataSync task, starting your task, or updating your task.
The following steps show how to configure a summary only task report when creating a task.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Tasks**, and then choose **Create task**.
1. Configure your task's source and destination locations.
For more information, see [Where can I transfer my data with AWS DataSync?](working-with-locations.md)
1. Scroll down to the **Task report** section. For **Report type**, choose **Summary only**.
1. For **S3 bucket for reports**, choose an S3 bucket where you want DataSync to upload your task report.
**Tip**
If you're planning to transfer data to an S3 bucket, don't use the same bucket for your task report if you [disable the **Keep deleted files** option](configure-metadata.md). Otherwise, DataSync will delete any previous task reports each time you execute a task since those reports don't exist in your source location.
1. For **Folder**, enter a prefix to use for your task report when DataSync uploads the report to your S3 bucket (for example, **reports/**).
Make sure to include the appropriate delimiter character at the end of your prefix. This character is usually a forward slash (`/`). For more information, see [Organizing objects by using prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) in the *Amazon S3 User Guide*.
1. For **IAM role**, do one of the following:
+ Choose **Autogenerate** to have DataSync automatically create an IAM role with the permissions that are required to access the S3 bucket.
If DataSync previously created an IAM role for this S3 bucket, that role is chosen by default.
+ Choose a custom IAM role that you created.
In some cases, you might need to create the role yourself. For more information, see [Allow DataSync to upload task reports to your S3 bucket](#task-report-access).
**Important**
If your S3 bucket is encrypted with a customer managed SSE-KMS key, the key's policy must include the IAM role that DataSync uses to access the bucket.
For more information, see [Accessing S3 buckets using server-side encryption](create-s3-location.md#create-s3-location-encryption).
1. Finish creating your task, and then [start the task](run-task.md) to begin transferring your data.
When your transfer is complete, you can [view your task report](task-report-viewing.md).
### Using the AWS CLI
1. Copy the following `create-task` AWS Command Line Interface (AWS CLI) command:
```
aws datasync create-task \
--source-location-arn arn:aws:datasync:us-east-1:123456789012:location/loc-12345678abcdefgh \
--destination-location-arn arn:aws:datasync:us-east-1:123456789012:location/loc-abcdefgh12345678 \
--task-report-config '{
"Destination":{
"S3":{
"Subdirectory":"reports/",
"S3BucketArn":"arn:aws:s3:::your-task-reports-bucket",
"BucketAccessRoleArn":"arn:aws:iam::123456789012:role/bucket-iam-role"
}
},
"OutputType":"SUMMARY_ONLY"
}'
```
1. For the `--source-location-arn` parameter, specify the Amazon Resource Name (ARN) of the source location in your transfer. Replace `us-east-1` with the appropriate AWS Region, replace `123456789012` with the appropriate AWS account number, and replace `12345678abcdefgh` with the appropriate source location ID.
1. For the `--destination-location-arn` parameter, specify the ARN of the destination location in your transfer. Replace `us-east-1` with the appropriate AWS Region, replace `123456789012` with the appropriate AWS account number, and replace `abcdefgh12345678` with the appropriate destination location ID.
1. For the `--task-report-config` parameter, do the following:
+ `Subdirectory` – Replace `reports/` with the prefix in your S3 bucket where you want DataSync to upload your task reports.
Make sure to include the appropriate delimiter character at the end of your prefix. This character is usually a forward slash (`/`). For more information, see [Organizing objects by using prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) in the *Amazon S3 User Guide*.
+ `S3BucketArn` – Specify the ARN of the S3 bucket where you want to upload your task report.
**Tip**
If you're planning to transfer data to an S3 bucket, don't use the same bucket for your task report if you [disable the **Keep deleted files** option](configure-metadata.md). Otherwise, DataSync will delete any previous task reports each time you execute a task since those reports don't exist in your source location.
+ `BucketAccessRoleArn` – Specify the IAM role that allows DataSync to upload a task report to your S3 bucket.
For more information, see [Allow DataSync to upload task reports to your S3 bucket](#task-report-access).
**Important**
If your S3 bucket is encrypted with a customer managed SSE-KMS key, the key's policy must include the IAM role that DataSync uses to access the bucket.
For more information, see [Accessing S3 buckets using server-side encryption](create-s3-location.md#create-s3-location-encryption).
+ `OutputType` – Specify `SUMMARY_ONLY`.
For more information, see [Summary only task reports](task-reports.md#task-report-types-summary).
1. Run the `create-task` command to create your task.
You get a response like the following that shows you the ARN of the task that you created. You will need this ARN to run the `start-task-execution` command.
```
{
"TaskArn": "arn:aws:datasync:us-east-1:123456789012:task/task-12345678abcdefgh"
}
```
1. Copy the following `start-task-execution` command.
```
aws datasync-task-report start-task-execution \
--task-arn arn:aws:datasync:us-east-1:123456789012:task/task-12345678abcdefgh
```
1. For the `--task-arn` parameter, specify the ARN of the task that you're starting. Use the ARN that you received from running the `create-task` command.
1. Run the `start-task-execution` command.
When your transfer is complete, you can [view your task report](task-report-viewing.md).
## Creating a standard task report
You can configure a [standard task report](task-reports.md#task-report-types-standard) when creating your DataSync task, starting your task, or updating your task.
The following steps show how to configure a standard task report when creating a task.
### Using the DataSync console
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Tasks**, and then choose **Create task**.
1. Configure your task's source and destination locations.
For more information, see [Where can I transfer my data with AWS DataSync?](working-with-locations.md)
1. Scroll down to the **Task report** section. For **Report type**, choose **Standard report**.
1. For **Report level**, choose one of the following:
+ **Errors only** – Your task report includes only issues with what DataSync tried to transfer, skip, verify, and delete.
+ **Successes and errors** – Your task report includes what DataSync successfully transferred, skipped, verified, and deleted and what it didn't.
+ **Custom** – Allows you to choose whether you want to see errors only or successes and errors for specific aspects of your task report.
For example, you can choose **Successes and errors** for the transferred files list but **Errors only** for the rest of the report.
1. If you're transferring to an S3 bucket that uses object versioning, keep **Include Amazon S3 object versions** selected if you want your report to include the new version for each transferred object.
1. For **S3 bucket for reports**, choose an S3 bucket where you want DataSync to upload your task report.
**Tip**
If you're planning to transfer data to an S3 bucket, don't use the same bucket for your task report if you [disable the **Keep deleted files** option](configure-metadata.md). Otherwise, DataSync will delete any previous task reports each time you execute a task since those reports don't exist in your source location.
1. For **Folder**, enter a prefix to use for your task report when DataSync uploads the report to your S3 bucket (for example, **reports/**). Make sure to include the appropriate delimiter character at the end of your prefix. This character is usually a forward slash (`/`). For more information, see [Organizing objects by using prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) in the *Amazon S3 User Guide*.
1. For **IAM role**, do one of the following:
+ Choose **Autogenerate** to have DataSync automatically create an IAM role with the permissions that are required to access the S3 bucket.
If DataSync previously created an IAM role for this S3 bucket, that role is chosen by default.
+ Choose a custom IAM role that you created.
In some cases, you might need to create the role yourself. For more information, see [Allow DataSync to upload task reports to your S3 bucket](#task-report-access).
**Important**
If your S3 bucket is encrypted with a customer managed SSE-KMS key, the key's policy must include the IAM role that DataSync uses to access the bucket.
For more information, see [Accessing S3 buckets using server-side encryption](create-s3-location.md#create-s3-location-encryption).
1. Finish creating your task and [start the task](run-task.md) to begin transferring your data.
When your transfer is complete, you can [view your task report](task-report-viewing.md).
### Using the AWS CLI
1. Copy the following `create-task` command:
```
aws datasync create-task \
--source-location-arn arn:aws:datasync:us-east-1:123456789012:location/loc-12345678abcdefgh \
--destination-location-arn arn:aws:datasync:us-east-1:123456789012:location/loc-abcdefgh12345678 \
--task-report-config '{
"Destination":{
"S3":{
"Subdirectory":"reports/",
"S3BucketArn":"arn:aws:s3:::your-task-reports-bucket",
"BucketAccessRoleArn":"arn:aws:iam::123456789012:role/bucket-iam-role"
}
},
"OutputType":"STANDARD",
"ReportLevel":"level-of-detail",
"ObjectVersionIds":"include-or-not"
}'
```
1. For the `--source-location-arn` parameter, specify the ARN of the source location in your transfer. Replace `us-east-1` with the appropriate AWS Region, replace `123456789012` with the appropriate AWS account number, and replace `12345678abcdefgh` with the appropriate source location ID.
1. For the `--destination-location-arn` parameter, specify the ARN of the destination location in your transfer. Replace `us-east-1` with the appropriate AWS Region, replace `123456789012` with the appropriate AWS account number, and replace `abcdefgh12345678` with the appropriate destination location ID.
1. For the `--task-report-config` parameter, do the following:
+ `Subdirectory` – Replace `reports/` with the prefix in your S3 bucket where you want DataSync to upload your task reports. Make sure to include the appropriate delimiter character at the end of your prefix. This character is usually a forward slash (`/`). For more information, see [Organizing objects by using prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) in the *Amazon S3 User Guide*.
+ `S3BucketArn` – Specify the ARN of the S3 bucket where you want to upload your task report.
**Tip**
If you're planning to transfer data to an S3 bucket, don't use the same bucket for your task report if you [disable the **Keep deleted files** option](configure-metadata.md). Otherwise, DataSync will delete any previous task reports each time you execute a task since those reports don't exist in your source location.
+ `BucketAccessRoleArn` – Specify the IAM role that allows DataSync to upload a task report to your S3 bucket.
For more information, see [Allow DataSync to upload task reports to your S3 bucket](#task-report-access).
**Important**
If your S3 bucket is encrypted with a customer managed SSE-KMS key, the key's policy must include the IAM role that DataSync uses to access the bucket.
For more information, see [Accessing S3 buckets using server-side encryption](create-s3-location.md#create-s3-location-encryption).
+ `OutputType` – Specify `STANDARD` report.
For more information, see [Standard task reports](task-reports.md#task-report-types-standard)Types of task reports.
+ (Optional) `ReportLevel` – Specify whether you want `ERRORS_ONLY` (the default) or `SUCCESSES_AND_ERRORS` in your report.
+ (Optional) `ObjectVersionIds` – If you're transferring to an S3 bucket that uses object versioning, specify `NONE` if you don't want to include the new version for each transferred object in the report.
By default, this option is set to `INCLUDE`.
+ (Optional) `Overrides` – Customize the `ReportLevel` of a particular aspect of your report.
For example, you might want to see `SUCCESSES_AND_ERRORS` for the list of what DataSync deletes in your destination location, but you want `ERRORS_ONLY` for everything else. In this example, you would add the following `Overrides` option to the `--task-report-config` parameter:
```
"Overrides":{
"Deleted":{
"ReportLevel":"SUCCESSES_AND_ERRORS"
}
}
```
If you don't use `Overrides`, your entire report uses the `ReportLevel` that you specify.
1. Run the `create-task` command to create your task.
You get a response like the following that shows you the ARN of the task that you created. You will need this ARN to run the `start-task-execution` command.
```
{
"TaskArn": "arn:aws:datasync:us-east-1:123456789012:task/task-12345678abcdefgh"
}
```
1. Copy the following `start-task-execution` command.
```
aws datasync-task-report start-task-execution \
--task-arn arn:aws:datasync:us-east-1:123456789012:task/task-12345678abcdefgh
```
1. For the `--task-arn` parameter, specify the ARN of the task you're running. Use the ARN that you received from running the `create-task` command.
1. Run the `start-task-execution` command.
When your transfer is complete, you can [view your task report](task-report-viewing.md).
# Viewing your DataSync task reports
DataSync creates task reports for every task execution. When your execution completes, you can find the related task reports in your S3 bucket. Task reports are organized under prefixes that include the IDs of your tasks and their executions.
To help locate task reports in your S3 bucket, use these examples:
+ **Summary only task report** – `reports-prefix/Summary-Reports/task-id-folder/task-execution-id-folder`
+ **Standard task report** – `reports-prefix/Detailed-Reports/task-id-folder/task-execution-id-folder`
Because task reports are in JSON format, you have several options for viewing your reports:
+ View a report by using [Amazon S3 Select](https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html).
+ Visualize reports by using AWS services such as AWS Glue, Amazon Athena, and Amazon Quick. For more information about visualizing your task reports, see the [AWS Storage Blog](https://aws.amazon.com/blogs/storage/derive-insights-from-aws-datasync-task-reports-using-aws-glue-amazon-athena-and-amazon-quicksight/).
# Monitoring data transfers with Amazon CloudWatch Logs
You can monitor your AWS DataSync transfer by using CloudWatch Logs. We recommend that you configure your task to at least log basic information (such as transfer errors).
## Allowing DataSync to upload logs to a CloudWatch log group
To [configure logging](#configure-logging-for-task) for your DataSync task, you need a CloudWatch log group that DataSync has permission to send logs to. You set up this access through an AWS Identity and Access Management (IAM) role. How this specifically works depends on your [task mode](choosing-task-mode.md).
------
#### [ Enhanced mode ]
With Enhanced mode, DataSync automatically sends task logs to a log group named `/aws/datasync`. If that log group doesn't exist in your AWS Region, DataSync creates the log group on your behalf by using an IAM [service-linked role](https://docs.aws.amazon.com/datasync/latest/userguide/using-service-linked-roles-service-action-2.html) when you create your task.
------
#### [ Basic mode ]
There are a couple ways to set up a CloudWatch log group for a DataSync task using Basic mode. In the console, you can automatically create an IAM role that in most cases includes the permissions that DataSync requires to upload logs. Keep in mind that this automatically generated role might not meet your needs from a least-privilege standpoint.
If you want to use an existing CloudWatch log group or are creating your tasks programmatically, you must create the IAM role yourself.
The following example is an IAM policy that grants these permissions.
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "DataSyncLogsToCloudWatchLogs",
"Effect": "Allow",
"Action": [
"logs:PutLogEvents",
"logs:CreateLogStream"
],
"Principal": {
"Service": "datasync.amazonaws.com"
},
"Condition": {
"ArnLike": {
"aws:SourceArn": [
"arn:aws:datasync:us-east-1:444455556666:task/*"
]
},
"StringEquals": {
"aws:SourceAccount": "444455556666"
}
},
"Resource": "arn:aws:logs:us-east-1:444455556666:log-group:*:*"
}
]
}
```
The policy uses `Condition` statements to help ensure that only DataSync tasks from the specified account have access to the specified CloudWatch log group. We recommend using the [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn) and [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) global condition context keys in these `Condition` statements to protect against the confused deputy problem. For more information, see [Cross-service confused deputy prevention](cross-service-confused-deputy-prevention.md).
To specify the DataSync task or tasks, replace *`region`* with the Region code for the AWS Region where the tasks are located (for example, `us-west-2`), and replace *`account-id`* with the AWS account ID of the account that contains the tasks. To specify the CloudWatch log group, replace the same values. You can also modify the `Resource` statement to target specific log groups. For more information about using `SourceArn` and `SourceAccount`, see [Global condition keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) in the *IAM User Guide*.
To apply the policy, save this policy statement to a file on your local computer. Then run the following AWS CLI command to apply the resource policy. To use this example command, replace `full-path-to-policy-file` with the path to the file that contains your policy statement.
```
aws logs put-resource-policy --policy-name trust-datasync --policy-document file://full-path-to-policy-file
```
**Note**
Run this command by using the same AWS account and AWS Region where you activated your DataSync agent.
For more information, see the [https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html).
------
## Configuring logging for your DataSync task
We recommend that you configure at least some level of logging for your DataSync task.
**Before you begin**
DataSync needs permission to upload logs to a CloudWatch log group. For more information, see [Allowing DataSync to upload logs to a CloudWatch log group](#cloudwatchlogs).
### Using the DataSync console
The following instructions describe how to configure CloudWatch logging when creating a task. You also can configure logging when editing a task.
1. Open the AWS DataSync console at [https://console.aws.amazon.com/datasync/](https://console.aws.amazon.com/datasync/).
1. In the left navigation pane, expand **Data transfer**, then choose **Tasks**, and then choose **Create task**.
1. Configure your task's source and destination locations.
For more information, see [Where can I transfer my data with AWS DataSync?](working-with-locations.md)
1. On the **Configure settings** page, choose a [task mode](choosing-task-mode.md) and any other options.
You might be interested in some of the following options:
+ Specify what data to transfer by using a [manifest](transferring-with-manifest.md) or [filters](filtering.md).
+ Configure how to [handle file metadata](configure-metadata.md) and [verify data integrity](configure-data-verification-options.md).
1. For **Log level**, choose one of the following options:
+ **Log basic information such as transfer errors** – Publish logs with only basic information (such as transfer errors).
+ **Log all transferred objects and files** – Publish logs for all files or objects that DataSync transfers and performs data-integrity checks on.
+ **Don't generate logs**
1. Do one of the following depending on the task mode you're using to create or specify a CloudWatch log group:
------
#### [ Enhanced mode ]
When you choose **Create task**, DataSync automatically uses (or creates) a log group named `/aws/datasync`.
------
#### [ Basic mode ]
For **CloudWatch log group**, specify a log group that DataSync has permission to upload logs to by doing one of the following:
+ Choose **Autogenerate** to automatically create a log group that allows DataSync to upload logs to it.
+ Choose an existing log group in your current AWS Region.
If you choose an existing log group, make sure that [DataSync has permission](#cloudwatchlogs) to upload logs to that log group.
------
1. Choose **Create task**.
You're ready to [start your task](run-task.md).
### Using the AWS CLI
1. Copy the following `create-task` command:
```
aws datasync create-task \
--source-location-arn "arn:aws:datasync:us-east-1:account-id:location/location-id" \
--destination-location-arn "arn:aws:datasync:us-east-1:account-id:location/location-id" \
--task-mode "ENHANCED-or-BASIC" \
--name "task-name" \
--options '{"LogLevel": "log-level"}' \
--cloudwatch-log-group-arn "arn:aws:logs:us-east-1:account-id:log-group:log-group-name:*"
```
1. For `--source-location-arn`, specify the Amazon Resource Name (ARN) of your source location.
1. For `--destination-location-arn`, specify the ARN of your destination location.
If you're transferring across AWS Regions or accounts, make sure that the ARN includes the other Region or account ID.
1. For `--task-mode`, specify `ENHANCED` or `BASIC`.
1. (Recommended) For `--name`, specify a name for your task that you can remember.
1. For `LogLevel`, specify one of the following options:
+ `BASIC` – Publish logs with only basic information (such as transfer errors).
+ `TRANSFER` – Publish logs for all files or objects that DataSync transfers and performs data-integrity checks on.
+ `NONE` – Don't generate logs.
1. For -`-cloudwatch-log-group-arn`, specify the ARN of a CloudWatch log group.
**Important**
If your `--task-mode` is `ENHANCED`, you don't need to specify this option. For more information, see [Allowing DataSync to upload logs to a CloudWatch log group](#cloudwatchlogs).
1. Run the `create-task` command.
If the command is successful, you get a response that shows you the ARN of the task that you created. For example:
```
{
"TaskArn": "arn:aws:datasync:us-east-1:111222333444:task/task-08de6e6697796f026"
}
```
You're ready to [start your task](run-task.md).
### Using the DataSync API
You can configure CloudWatch logging for your task by using the `CloudWatchLogGroupArn` parameter with any of the following operations:
+ [https://docs.aws.amazon.com/datasync/latest/userguide/API_CreateTask.html](https://docs.aws.amazon.com/datasync/latest/userguide/API_CreateTask.html)
+ [https://docs.aws.amazon.com/datasync/latest/userguide/API_UpdateTask.html](https://docs.aws.amazon.com/datasync/latest/userguide/API_UpdateTask.html)
## Viewing DataSync task logs
When you [start your task](run-task.md), you can view the task execution's logs by using the CloudWatch console or AWS CLI (among other options). For more information, see the [https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html).
DataSync provides JSON-structured logs for Enhanced mode tasks. Basic mode tasks have unstructured logs. The following examples show how verification errors display in Enhanced mode logs compared to Basic mode logs.
------
#### [ Enhanced mode log example ]
```
{
"Action": "VERIFY",
"Source": {
"LocationId": "loc-abcdef01234567890",
"RelativePath": "directory1/directory2/file1.txt"
},
"Destination": {
"LocationId": "loc-05ab2fdc272204a5f",
"RelativePath": "directory1/directory2/file1.txt",
"Metadata": {
"Type": "Object",
"ContentSize": 66060288,
"LastModified": "2024-10-03T20:46:58Z",
"S3": {
"SystemMetadata": {
"ContentType": "binary/octet-stream",
"ETag": "\"1234abcd5678efgh9012ijkl3456mnop\"",
"ServerSideEncryption": "AES256"
},
"UserMetadata": {
"file-mtime": "1602647222/222919600"
},
"Tags": {}
}
}
},
"ErrorCode": "FileNotAtSource",
"ErrorDetail": "Verification failed due to file being present at the destination but not at the source"
}
```
------
#### [ Basic mode log example ]
```
[NOTICE] Verification failed > /directory1/directory2/file1.txt
[NOTICE] /directory1/directory2/file1.txt dstMeta: type=R mode=0755 uid=65534 gid=65534 size=8972938 atime=1728657659/0 mtime=1728657659/0 extAttrsHash=0
[NOTICE] dstHash: f9c2cca900301d38b0930367d8d587153154af467da0fdcf1bebc0848ec72c0d
```
------
# Logging AWS DataSync API calls with AWS CloudTrail
AWS DataSync is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in DataSync. CloudTrail captures all API calls for DataSync as events. The calls that are captured include calls from the DataSync console and code calls to DataSync API operations.
If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket, including events for AWS DataSync. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in **Event history**. Using the information collected by CloudTrail, you can determine the request that was made to AWS DataSync, the IP address from which the request was made, who made the request, when it was made, and additional details.
To learn more about CloudTrail, see the [AWS CloudTrail User Guide](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/).
## Working with DataSync information in CloudTrail
CloudTrail is enabled on your AWS account when you create the account. When activity occurs in AWS DataSync, that activity is recorded in a CloudTrail event along with other AWS service events in **Event history**. You can view, search, and download recent events in your AWS account. For more information, see [Viewing events with CloudTrail event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).
For an ongoing record of events in your AWS account, including events for AWS DataSync, create a trail. A *trail* enables CloudTrail to deliver log files to an Amazon S3 bucket. By default, when you create a trail in the console, the trail applies to all AWS Regions. The trail logs events from all AWS Regions in the same AWS partition and delivers the log files to the Amazon S3 bucket that you specify. Additionally, you can configure other AWS services to further analyze and act upon the event data collected in CloudTrail logs. For more information, see the following:
+ [Overview for creating a trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html)
+ [CloudTrail supported services and integrations](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-aws-service-specific-topics.html#cloudtrail-aws-service-specific-topics-integrations)
+ [Configuring Amazon SNS notifications for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-sns-notifications-for-cloudtrail.html)
+ [Receiving CloudTrail log files from multiple Regions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/receive-cloudtrail-log-files-from-multiple-regions.html) and [Receiving CloudTrail log files from multiple accounts](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html)
All DataSync actions are logged by CloudTrail. (For more information, see the DataSync [API reference](https://docs.aws.amazon.com/datasync/latest/userguide/API_Operations.html).)
For example, calls to the `CreateAgent`, `CreateTask`, and `ListLocations` operations generate entries in the CloudTrail log files.
Every event or log entry contains information about who generated the request. The identity information helps you determine the following:
+ Whether the request was made with root or AWS Identity and Access Management (IAM) credentials.
+ Whether the request was made with temporary security credentials for a role or federated user.
+ Whether the request was made by another AWS service.
For more information, see [CloudTrail userIdentity element](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html) in the *AWS CloudTrail User Guide. *
## Understanding DataSync log file entries
A trail is a configuration that enables delivery of events as log files to an Amazon S3 bucket that you specify. CloudTrail log files contain one or more log entries. An event represents a single request from any source and includes information about the requested action, the date and time of the action, the request parameters, and so on. CloudTrail log files aren't an ordered stack trace of the public API calls, so they don't appear in any specific order.
The following example shows a CloudTrail log entry that demonstrates the `CreateTask` operation.
```
{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "1234567890abcdef0",
"arn": "arn:aws:iam::123456789012:user/user1",
"accountId": "123456789012",
"accessKeyId": "access key",
"userName": "user1",
"sessionContext": {
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "2018-12-13T14:56:46Z"
}
},
"invokedBy": "signin.amazonaws.com"
},
"eventTime": "2018-12-13T14:57:02Z",
"eventSource": "datasync.amazonaws.com",
"eventName": "CreateTask",
"awsRegion": "ap-southeast-1",
"sourceIPAddress": "192.0.2.1",
"userAgent": "signin.amazonaws.com",
"requestParameters": {
"cloudWatchLogGroupArn": "arn:aws:logs:ap-southeast-1:123456789012:log-group:MyLogGroup",
"name": "MyTask-NTIzMzY1",
"tags": [],
"destinationLocationArn": "arn:aws:datasync:ap-southeast-1:123456789012:location/loc-abcdef01234567890",
"options": {
"bytesPerSecond": -1,
"verifyMode": "POINT_IN_TIME_CONSISTENT",
"uid": "INT_VALUE",
"posixPermissions": "PRESERVE",
"mtime": "PRESERVE",
"gid": "INT_VALUE",
"preserveDevices": "NONE",
"preserveDeletedFiles": "REMOVE",
"atime": "BEST_EFFORT"
},
"sourceLocationArn": "arn:aws:datasync:ap-southeast-1:123456789012:location/loc-021345abcdef6789"
},
"responseElements": {
"taskArn": "arn:aws:datasync:ap-southeast-1:123456789012:task/task-1234567890abcdef0"
},
"requestID": "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111",
"eventID": "a1b2c3d4-5678-90ab-cdef-EXAMPLE22222",
"eventType": "AwsApiCall",
"recipientAccountId": "123456789012"
}
```
# Monitoring events by using Amazon EventBridge
Amazon EventBridge events describe changes in DataSync resources. You can set up rules to match these events and route them to one or more target functions or streams. Events are emitted on a best-effort basis.
## DataSync transfer events
The following EventBridge events are available for DataSync transfers.
| **Agent state changes** |
| --- |
| Event | Description |
| Online | The agent is configured properly and ready to use. This is the normal running status for an agent. |
| Offline | The agent has been out of contact with the DataSync service for five minutes or longer. This can happen for a few reasons. For more information, see [What do I do if my agent is offline?](troubleshooting-datasync-agents.md#troubleshoot-agent-offline) |
| **Location state changes** |
| --- |
| Event | Description |
| Adding | DataSync is adding a location. |
| Available | The location is created and is available to use. |
| **Task state changes** |
| --- |
| Event | Description |
| Available | The task was created and is ready to start. |
| Running | The task is in progress and functioning properly. |
| Unavailable | The task isn't configured properly and can't be used. You might see this event when an agent associated with the task goes offline. |
| Queued | Another task is running and using the same agent. DataSync runs tasks in series (first in, first out). |
| **Task execution state changes** |
| --- |
| Event | Description |
| Queueing | Another task execution is running and using the same DataSync agent. For more information, see [Knowing when your task is queued](run-task.md#queue-task-execution). |
| Launching | DataSync is initializing the task execution. This status usually goes quickly but can take up to a few minutes. |
| Preparing | DataSync is determining what data to transfer. This step can take just minutes or a few hours depending on the number of files, objects, or directories in both locations and on how you configure your task. Preparation also might not be applicable to your task. For more information, see [How DataSync prepares your data transfer](how-datasync-transfer-works.md#how-datasync-prepares). |
| Transferring | DataSync is performing the actual data transfer. |
| Verifying | DataSync is performing a data-integrity check at the end of the transfer. |
| Success | The task execution succeeded. |
| Cancelling | The task execution is in the process of being cancelled. |
| Error | The task execution failed. |
# Monitoring AWS DataSync with manual tools
You can track your AWS DataSync transfers from the console or the command line.
## Monitoring your transfer by using the DataSync console
You can monitor your DataSync transfer by using the console, which provides real-time metrics such as data transferred, data and file throughput, and data compression.
**To monitor your transfer by using the DataSync console**
1. After you [start your DataSync task](run-task.md#starting-task), choose **See execution details**.
1. View metrics about your transfer.
## Monitoring your transfer by using the AWS CLI
You can monitor your DataSync transfer by using the AWS Command Line Interface (AWS CLI).
Copy the following `describe-task-execution` command. To use this example command, replace the `user input placeholders` with your own information.
```
aws datasync describe-task-execution \
--task-execution-arn 'arn:aws:datasync:region:account-id:task/task-id/execution/task-execution-id'
```
This command returns information about a task execution similar to that shown following.
```
{
"BytesCompressed": 3500,
"BytesTransferred": 5000,
"BytesWritten": 5000,
"EstimatedBytesToTransfer": 5000,
"EstimatedFilesToDelete": 10,
"EstimatedFilesToTransfer": 100,
"FilesDeleted": 10,
"FilesSkipped": 0,
"FilesTransferred": 100,
"FilesVerified": 100,
"Result": {
"ErrorCode": "??????",
"ErrorDetail": "??????",
"PrepareDuration": 100,
"PrepareStatus": "SUCCESS",
"TransferDuration": 60,
"TransferStatus": "AVAILABLE",
"VerifyDuration": 30,
"VerifyStatus": "SUCCESS"
},
"StartTime": 1532660733.39,
"Status": "SUCCESS",
"OverrideOptions": {
"Atime": "BEST_EFFORT",
"BytesPerSecond": "1000",
"Gid": "NONE",
"Mtime": "PRESERVE",
"PosixPermissions": "PRESERVE",
"PreserveDevices": "NONE",
"PreserveDeletedFiles": "PRESERVE",
"Uid": "NONE",
"VerifyMode": "POINT_IN_TIME_CONSISTENT"
},
"TaskExecutionArn": "arn:aws:datasync:us-east-1:111222333444:task/task-aaaabbbbccccddddf/execution/exec-1234abcd1234abcd1",
"TaskReportConfig": {
"Destination": {
"S3": {
"BucketAccessRoleArn": "arn:aws:iam::111222333444:role/my-datasync-role",
"S3BucketArn": "arn:aws:s3:::amzn-s3-demo-bucket/*",
"Subdirectory": "reports"
}
},
"ObjectVersionIds": "INCLUDE",
"OutputType": "STANDARD",
"Overrides": {
"Deleted": {
"ReportLevel": "ERRORS_ONLY"
},
"Skipped": {
"ReportLevel": "SUCCESSES_AND_ERRORS"
},
"Transferred": {
"ReportLevel": "ERRORS_ONLY"
},
"Verified": {
"ReportLevel": "ERRORS_ONLY"
}
},
"ReportLevel": "ERRORS_ONLY"
}
}
```
+ If the task execution succeeds, the value of **Status** changes to **SUCCESS**. For information about what the response elements mean, see [DescribeTaskExecution](API_DescribeTaskExecution.md).
+ If the task execution fails, the result sends error codes that can help you troubleshoot issues. For information about the error codes, see [TaskExecutionResultDetail](API_TaskExecutionResultDetail.md).
## Monitoring your transfer by using the `watch` utility
To monitor the progress of your task in real time from the command line, you can use the standard Unix `watch` utility. Task execution duration values are measured in milliseconds.
The `watch` utility doesn't recognize the DataSync alias. The following example shows how to invoke the CLI directly. To use this example command, replace the `user input placeholders` with your own information.
```
# pass '-n 1' to update every second and '-d' to highlight differences
$ watch -n 1 -d \ "aws datasync describe-task-execution --task-execution-arn 'arn:aws:datasync:region:account-id:task/task-id/execution/task execution-id'"
```