Using SFTP connectors and tracking their progress - AWS Transfer Family
Send and retrieve files by using an SFTP connectorTrack the progress of your file transfer requests

Using SFTP connectors and tracking their progress

SFTP connectors extend the capabilities of AWS Transfer Family to communicate with remote servers both in the cloud and on-premises. You can integrate data that's generated and stored in remote sources with your AWS hosted data warehouses for analytics, business applications, reporting, and auditing. To initiate a file transfer to a remote SFTP server, you use the StartFileTransfer API operation, which uses SFTP connectors to perform the transfer.

You can monitor your file transfers by running the ListFileTransferResults API operation, or by checking your server logs. Connector activity is logged to log streams that have the format of aws/transfer/connector-id, for example, aws/transfer/c-1234567890abcdef0. If you don't see any logs for your connector, make sure that you have specified a logging role with the correct permissions for your connector.

For details on creating connectors, see Configure SFTP connectors.

Note

Currently, SFTP connectors can only be used to connect to remote SFTP servers that offer an internet-accessible endpoint.

Send and retrieve files by using an SFTP connector

To send and retrieve files by using an SFTP connector, you use the StartFileTransfer API operation and specify the following parameters, depending on whether you're sending files (outbound transfers) or receiving files (inbound transfers). Note that each StartFileTransfer request can contain 10 distinct paths.

Note

By default, SFTP connectors process one file at a time, transferring files sequentially. You have an option to accelerate transfer performance by having your connectors create concurrent sessions with remote servers that support concurrent sessions from the same user, and process up to 5 files in parallel.

To enable concurrent connections for any connector, contact AWS Support at Contact AWS to submit a quota increase request to the AWS Transfer Family service team.

  • Outbound transfers

    • send-file-paths contains from one to ten source file paths, for files to transfer to the partner's SFTP server.

    • remote-directory-path is the remote path to send a file to on the customer's SFTP server.

  • Inbound transfers

    • retrieve-file-paths contains from one to ten remote paths. Each path specifies a location for transferring files from the partner's SFTP server to your Transfer Family server.

    • local-directory-path is the Amazon S3 location (bucket and optional prefix) where your files are stored.

To send files, you specify the send-file-paths and remote-directory-path parameters. You can specify up to 10 files for the send-file-paths parameter. The following example command sends the files named /amzn-s3-demo-source-bucket/file1.txt and /amzn-s3-demo-source-bucket/file2.txt, located in Amazon S3 storage, to the /tmp directory on your partner's SFTP server. To use this example command, replace the amzn-s3-demo-source-bucket with your own bucket.

aws transfer start-file-transfer --send-file-paths /amzn-s3-demo-source-bucket/file1.txt /amzn-s3-demo-source-bucket/file2.txt \
    --remote-directory-path /tmp --connector-id c-1111AAAA2222BBBB3 --region us-east-2

To retrieve files, you specify the retrieve-file-paths and local-directory-path parameters. The following example retrieves the files /my/remote/file1.txt and /my/remote/file2.txt on the partner's SFTP server, and places it in the Amazon S3 location /amzn-s3-demo-bucket/prefix. To use this example command, replace the user input placeholders with your own information.

aws transfer start-file-transfer --retrieve-file-paths /my/remote/file1.txt  /my/remote/file2.txt \
   --local-directory-path /amzn-s3-demo-bucket/prefix --connector-id c-2222BBBB3333CCCC4 --region us-east-2

The previous examples specify absolute paths on the SFTP server. You can also use relative paths: that is, paths that are relative to the SFTP user's home directory. For example, if the SFTP user is marymajor and their home directory on the SFTP server is /users/marymajor/, the following command sends /amzn-s3-demo-source-bucket/file1.txt to /users/marymajor/test-connectors/file1.txt

aws transfer start-file-transfer --send-file-paths /amzn-s3-demo-source-bucket/file1.txt \
   --remote-directory-path test-connectors --connector-id c-2222BBBB3333CCCC4 --region us-east-2

Track the progress of your file transfer requests

To track the progress of a file transfer operation, you use the ListFileTransferResults API operation, which returns real-time updates and detailed information on the status of each individual file being transferred in a specific file transfer operation. You specify the file transfer by providing its Connector ID and its Transfer ID. The following example returns a list of files for connector ID a-11112222333344444 and transfer-ID aa1b2c3d4-5678-90ab-cdef-EXAMPLE11111.

aws transfer list-file-transfer-results --connector-id a-11112222333344444 --transfer-id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111
Note

File transfer results are available up to 7 days after you call the ListFileTransferResults API operation.