Amazon S3 on Outposts with local Amazon EMR on Outposts

Amazon EMR is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark, on AWS to process and analyze vast amounts of data. By using these frameworks and related open-source projects, you can process data for analytics purposes and business intelligence workloads. Amazon EMR also helps you transform and move large amounts of data into and out of other AWS data stores and databases, and supports Amazon S3 on Outposts. For more information about Amazon EMR, see Amazon EMR on Outposts in the Amazon EMR Management Guide.

For Amazon S3 on Outposts, Amazon EMR started to support the Apache Hadoop S3A connector in version 7.0.0. Earlier versions of Amazon EMR don't support local S3 on Outposts, and the EMR File System (EMRFS) is not supported.

Supported applications

Amazon EMR with Amazon S3 on Outposts supports the following applications:

For more information, see the Amazon EMR Release Guide.

Create and configure an Amazon S3 on Outposts bucket

Amazon EMR uses the AWS SDK for Java with Amazon S3 on Outposts to store input data and output data. Your Amazon EMR log files are stored in a Regional Amazon S3 location that you select and aren't stored locally on the Outpost. For more information, see Amazon EMR logs in the Amazon EMR Management Guide.

To conform with Amazon S3 and DNS requirements, S3 on Outposts buckets have naming restrictions and limitations. For more information, see Creating an S3 on Outposts bucket.

With Amazon EMR version 7.0.0 and later, you can use Amazon EMR with S3 on Outposts and the S3A file system.

Prerequisites

S3 on Outposts permissions – When you create your Amazon EMR instance profile, your role must contain the AWS Identity and Access Management (IAM) namespace for S3 on Outposts. S3 on Outposts has its own namespace, s3-outposts*. For an example policy that uses this namespace, see Setting up IAM with S3 on Outposts.

S3A connector – To configure your EMR cluster to access data from an Amazon S3 on Outposts bucket, you must use the Apache Hadoop S3A connector. To use the connector, ensure that all of your S3 URIs use the s3a scheme. If they don't, you can configure the file system implementation that you use for your EMR cluster so that your S3 URIs work with the S3A connector.

To configure the file system implementation to work with the S3A connector, you use the fs.file_scheme.impl and fs.AbstractFileSystem.file_scheme.impl configuration properties for your EMR cluster, where file_scheme corresponds to the type of S3 URIs that you have. To use the following example, replace the user input placeholders with your own information. For example, to change the file system implementation for S3 URIs that use the s3 scheme, specify the following cluster configuration properties:

[
  {
"Classification": "core-site",
    "Properties": {
    "fs.s3.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem",
    "fs.AbstractFileSystem.s3.impl": "org.apache.hadoop.fs.s3a.S3A"
    }
  }
]
    

To use S3A, set the fs.file_scheme.impl configuration property to org.apache.hadoop.fs.s3a.S3AFileSystem, and set the fs.AbstractFileSystem.file_scheme.impl property to org.apache.hadoop.fs.s3a.S3A.

For example, if you are accessing the path s3a://bucket/..., set the fs.s3a.impl property to org.apache.hadoop.fs.s3a.S3AFileSystem, and set the fs.AbstractFileSystem.s3a.impl property to org.apache.hadoop.fs.s3a.S3A.

Getting started using Amazon EMR with Amazon S3 on Outposts

The following topics explain how to get started using Amazon EMR with Amazon S3 on Outposts.

Create a permissions policy

Before you can create an EMR cluster that uses Amazon S3 on Outposts, you must create an IAM policy to attach to the Amazon EC2 instance profile for the cluster. The policy must have permissions to access the S3 on Outposts access point Amazon Resource Name (ARN). For more information about creating IAM policies for S3 on Outposts, see Setting up IAM with S3 on Outposts.

The following example policy shows how to grant the required permissions. After you create the policy, attach the policy to the instance profile role that you use to create your EMR cluster, as described in the Create and configure your cluster section. To use this example, replace the user input placeholders with your own information.

{
"Version":"2012-10-17", 
  "Statement": [
        {
  "Effect": "Allow",
            "Resource": "arn:aws:s3-outposts:us-west-2:111122223333:outpost/op-01ac5d28a6a232904/accesspoint/access-point-name,
            "Action": [
                "s3-outposts:*"
            ]
        }
    ]
    
 }

Create and configure your cluster

To create a cluster that runs Spark with S3 on Outposts, complete the following steps in the console.

Configurations overview

The following table describes S3A configurations and the values to specify for their parameters when you set up a cluster that uses S3 on Outposts with Amazon EMR.

The following table describes Spark configurations and the values to specify for their parameters when you set up a cluster that uses S3 on Outposts with Amazon EMR.

Considerations

Consider the following when you integrate Amazon EMR with S3 on Outposts buckets: