AWS Transfer Family SFTP connectors establish a relationship for sending files and messages between Amazon storage and an external partner, using the SFTP protocol. You can send files from Amazon S3 to an external, partner-owned destination. You can also use an SFTP connector to retrieve files from a partner's SFTP server.
Note
Currently, SFTP connectors can only be used to connect to remote SFTP servers that offer an internet-accessible endpoint.
The following video provides a brief introduction to Transfer Family SFTP connectors.
Topics
Quotas for SFTP
connectors
The following quotas are in place for SFTP connectors.
Note
More service quotas for SFTP connectors are listed in AWS Transfer Family endpoints and quotas in the Amazon Web Services General Reference.
Name | Default | Adjustable |
---|---|---|
Maximum test connection transactions per second (TPS) | 1 request per second, per account | No |
Maximum queue size for pending file transfers | 1000 | No |
Maximum file size | 150 gibibytes (GiB) | No |
Maximum transfer time per file | 12 hours | No |
Maximum request wait time per file | 6 hours | No |
Maximum bandwidth for connectors per account (both SFTP and AS2 connectors contribute to this value) | 50 MBps | No |
Maximum number of items for directory listing operations | 10,000 | No |
Maximum number of files per StartFileTransfer
request |
10 | No |
Note
By default, SFTP connectors process one file at a time, transferring files sequentially. You have an option to accelerate transfer performance by having your connectors create concurrent sessions with remote servers that support concurrent sessions from the same user, and process up to 5 files in parallel.
To enable concurrent connections for any connector, contact AWS Support at
Contact AWS
For storing the credentials for SFTP connectors, there are quotas associated with each Secrets Manager secret. If you use the same secret to store multiple types of keys, for multiple purposes, you may encounter these quotas.
-
Total length for a single secret: 12,000 characters
-
Maximum length of the
Password
string: 1024 characters -
Maximum length of the
PrivateKey
string: 8192 characters -
Maximum length of the
Username
string: 100 characters
Scaling your SFTP
connectors
This section describes considerations for how to scale your AWS Transfer Family SFTP connector workloads. You need to take into account the following three quotas that apply when you want to scale your workloads with SFTP connectors.
-
The maximum queue size. This refers to the maximum number of pending operations in a connector’s queue that have been requested. A pending operation refers to any previously submitted transfer request that has not yet completed, either successfully or unsuccessfully.
The maximum queue depth for pending requests is currently set at 1,000 per connector (as defined in AWS Transfer Family service quotas). Your workloads may exceed this service limit when you request thousands of transfer operations over a short duration, and you will receive a
ThrottlingException
with the messageExceeded maximum pending requests.
If your workloads are subject to this quota, contact the Transfer Family service team via AWS Support or your account team to discuss your scalability requirements.You can also take either or both of the following actions.
-
Distribute your file volumes across multiple connectors.
-
Have your connectors create concurrent sessions with the remote server to process multiple requests from the queue in parallel.
-
-
The number of concurrent sessions. By default, an SFTP connector transfers one file at a time, transferring files sequentially from its queue.
You have an option to accelerate transfer performance by having have your connectors transfer multiple files in parallel. You can create concurrent sessions with remote servers that support concurrent sessions from the same user, and process up to 5 files in parallel. To enable concurrent connections for any connector, contact AWS Support to submit a quota increase request to the AWS Transfer Family service team.
-
The rate of
StartFileTransfer
requests. You can request up to 100 file paths per second for transfer with each SFTP connector. The requested file paths are added to your connectors’ queue for processing. You can use theStartFileTransfer
command recursively to request up to 100 file paths per second per connector, irrespective of the number of files provided in an individualStartFileTransfer
command.
Using your connector
On the page where you can view the details for a connector, Transfer Family displays some useful example commands.
To view connector details
-
Open the AWS Transfer Family console at https://console.aws.amazon.com/transfer/
. -
In the left navigation pane, choose Connectors.
-
Choose the identifier in the Connector ID column to see the details page for the selected connector.
The following example commands are listed.
-
Command to test connectivity
aws transfer test-connection --region us-east-1 --connector-id c-1111aaaa2222bbbb3
-
Command to list files from a remote SFTP server
aws transfer start-directory-listing --region us-east-1 --connector-id c-1111aaaa2222bbbb3 --remote-directory-path /RemoteDir \ --output-directory-path /amzn-s3-demo-destination-bucket/prefix
-
Command to retrieve files from a remote SFTP server
aws transfer start-file-transfer --region us-east-1 --connector-id c-1111aaaa2222bbbb3 --retrieve-file-paths /RemoteDir/file1.txt \ --local-directory-path /amzn-s3-demo-destination-bucket/prefix
-
Command to send a file from Amazon S3 to a remote SFTP server
aws transfer start-file-transfer --region us-east-1 --connector-id c-1111aaaa2222bbbb3 \ --send-file-paths /amzn-s3-demo-source-bucket/file1.txt --remote-directory-path /RemoteDir
-
Command to query the real-time status of a file transfer operation using the specified connector
aws transfer list-file-transfer-results --region us-east-1 --connector-id c-1111aaaa2222bbbb3 \ --transfer-id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111
Note that you can easily copy the example commands to the clipboard, by selecting the corresponding Copy to clipboard button.
