从亚马逊简单存储服务 (Amazon S3) 存储桶下载 S3 “目录”

此示例说明如何使用TransferManager中的 S3 从 Amazon S3 存储桶下载 “目录”。 AWS SDK for Java 2.x 它还演示了如何在请求DownloadFilters中使用。

    /**
     * For standard buckets, S3 provides the illusion of a directory structure through the use of keys. When you upload
     * an object to an S3 bucket, you specify a key, which is essentially the "path" to the object. The key can contain
     * forward slashes ("/") to make it appear as if the object is stored in a directory structure, but this is just a
     * logical representation, not an actual directory.
     * <p><pre>
     * In this example, our S3 bucket contains the following objects:
     *
     * folder1/file1.txt
     * folder1/file2.txt
     * folder1/file3.txt
     * folder2/file1.txt
     * folder2/file2.txt
     * folder2/file3.txt
     * folder3/file1.txt
     * folder3/file2.txt
     * folder3/file3.txt
     *
     * When method `downloadS3Directories` is invoked with
     * `destinationPathURI` set to `/test`, the downloaded
     * directory looks like:
     *
     * |- test
     *    |- folder1
     *    	  |- file1.txt
     *    	  |- file2.txt
     *    	  |- file3.txt
     *    |- folder3
     *    	  |- file1.txt
     *    	  |- file2.txt
     *    	  |- file3.txt
     * </pre>
     *
     * @param transferManager    An S3TransferManager instance.
     * @param destinationPathURI local directory to hold the downloaded S3 'directories' and files.
     * @param bucketName         The S3 bucket that contains the 'directories' to download.
     * @return The number of objects (files, in this case) that were downloaded.
     */
    public Integer downloadS3Directories(S3TransferManager transferManager,
                                         URI destinationPathURI, String bucketName) {

        // Define the filters for which 'directories' we want to download.
        DownloadFilter folder1Filter = (S3Object s3Object) -> s3Object.key().startsWith("folder1/");
        DownloadFilter folder3Filter = (S3Object s3Object) -> s3Object.key().startsWith("folder3/");
        DownloadFilter folderFilter = s3Object -> folder1Filter.or(folder3Filter).test(s3Object);

        DirectoryDownload directoryDownload = transferManager.downloadDirectory(DownloadDirectoryRequest.builder()
                .destination(Paths.get(destinationPathURI))
                .bucket(bucketName)
                .filter(folderFilter)
                .build());
        CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

        Integer numFilesInFolder1 = Paths.get(destinationPathURI).resolve("folder1").toFile().list().length;
        Integer numFilesInFolder3 = Paths.get(destinationPathURI).resolve("folder3").toFile().list().length;

        try {
            assert numFilesInFolder1 == 3;
            assert numFilesInFolder3 == 3;
            assert !Paths.get(destinationPathURI).resolve("folder2").toFile().exists(); // `folder2` was not downloaded.
        } catch (AssertionError e) {
            logger.error("An assertion failed.");
        }

        completedDirectoryDownload.failedTransfers()
                .forEach(fail -> logger.warn("Object failed to transfer  [{}]", fail.exception().getMessage()));
        return numFilesInFolder1 + numFilesInFolder3;
    }