Using the AWS Glue Data Catalog as the metastore for Hive

Using Amazon EMR release 5.8.0 or later, you can configure Hive to use the AWS Glue Data Catalog as its metastore. We recommend this configuration when you require a persistent metastore or a metastore shared by different clusters, services, applications, or AWS accounts.

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores. The AWS Glue Data Catalog provides a unified metadata repository across a variety of data sources and data formats, integrating with Amazon EMR as well as Amazon RDS, Amazon Redshift, Redshift Spectrum, Athena, and any application compatible with the Apache Hive metastore. AWS Glue crawlers can automatically infer schema from source data in Amazon S3 and store the associated metadata in the Data Catalog. For more information about the Data Catalog, see Populating the AWS Glue Data Catalog in the AWS Glue Developer Guide.

Separate charges apply for AWS Glue. There is a monthly rate for storing and accessing the metadata in the Data Catalog, an hourly rate billed per minute for AWS Glue ETL jobs and crawler runtime, and an hourly rate billed per minute for each provisioned development endpoint. The Data Catalog allows you to store up to a million objects at no charge. If you store more than a million objects, you are charged USD$1 for each 100,000 objects over a million. An object in the Data Catalog is a table, partition, or database. For more information, see Glue Pricing.

Specifying AWS Glue Data Catalog as the metastore

You can specify the AWS Glue Data Catalog as the metastore using the AWS Management Console, AWS CLI, or Amazon EMR API. When you use the CLI or API, you use the configuration classification for Hive to specify the Data Catalog. In addition, with Amazon EMR 5.16.0 and later, you can use the configuration classification to specify a Data Catalog in a different AWS account. When you use the console, you can specify the Data Catalog using Advanced Options or Quick Options.

Console

To specify AWS Glue Data Catalog as the Hive metastore with the console

1. Sign in to the AWS Management Console, and open the Amazon EMR console at https://console.aws.amazon.com/emr.

2. Under EMR on EC2 in the left navigation pane, choose Clusters, and then choose Create cluster.

3. Under Application bundle, choose Core Hadoop, HBase, or Custom. If you customize your cluster, make sure that you select Hive or HCatalog as one of your applications.

4. Under AWS Glue Data Catalog settings, select the Use for Hive table metadata check box.

5. Choose any other options that apply to your cluster.

6. To launch your cluster, choose Create cluster.

CLI

To specify the AWS Glue Data Catalog as the Hive metastore with the AWS CLI

For more information about specifying a configuration classification using the AWS CLI and EMR API, see Configure applications.

• Specify the value for hive.metastore.client.factory.class using the hive-site configuration classification as shown in the following example:

  [
    {
      "Classification": "hive-site",
      "Properties": {
        "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
      }
    }
  ]
  						

  On EMR release versions 5.28.0, 5.28.1, 5.29.0, or 6.x, if you're creating a cluster using the AWS Glue Data Catalog as the metastore, set the hive.metastore.schema.verification to false. This prevents Hive and HCatalog from validating the metastore schema against MySQL. Without this configuration, the primary instance group will become suspended after reconfiguration on Hive or HCatalog.

  [
    {
      "Classification": "hive-site",
      "Properties": {
        "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
        "hive.metastore.schema.verification": "false"
      }
    }
  ]
  					

  If you already have a cluster on EMR release version 5.28.0, 5.28.1, or 5.29.0, you can set the primary instance group hive.metastore.schema.verification to false with following information:

     
      Classification = hive-site
      Property       = hive.metastore.schema.verification
      Value          = false
  					

  To specify a Data Catalog in a different AWS account, add the hive.metastore.glue.catalogid property as shown in the following example. Replace acct-id with the AWS account of the Data Catalog.

  [
    {
      "Classification": "hive-site",
      "Properties": {
        "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
        "hive.metastore.schema.verification": "false",
        "hive.metastore.glue.catalogid": "acct-id"
      }
    }
  ]
  				

IAM permissions

The EC2 instance profile for a cluster must have IAM permissions for AWS Glue actions. In addition, if you enable encryption for AWS Glue Data Catalog objects, the role must also be allowed to encrypt, decrypt and generate the AWS KMS key used for encryption.

Permissions for AWS Glue actions

If you use the default EC2 instance profile for Amazon EMR, no action is required. The AmazonElasticMapReduceforEC2Role managed policy that is attached to the EMR_EC2_DefaultRole allows all necessary AWS Glue actions. However, if you specify a custom EC2 instance profile and permissions, you must configure the appropriate AWS Glue actions. Use the AmazonElasticMapReduceforEC2Role managed policy as a starting point. For more information, see Service role for cluster EC2 instances (EC2 instance profile) in the Amazon EMR Management Guide.

Permissions for encrypting and decrypting AWS Glue Data Catalog

Your instance profile needs permission to encrypt and decrypt data using your key. You do not need to configure these permissions if both of the following statements apply:

Otherwise, you must add the following statement to the permissions policy attached to your EC2 instance profile.

[
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "kms:Decrypt",
                    "kms:Encrypt",
                    "kms:GenerateDataKey"
                ],
                "Resource": "arn:aws:kms:region:acct-id:key/12345678-1234-1234-1234-123456789012"
            }
        ]
    }
]

For more information about AWS Glue Data Catalog encryption, see Encrypting your data catalog in the AWS Glue Developer Guide.

Resource-based permissions

If you use AWS Glue in conjunction with Hive, Spark, or Presto in Amazon EMR, AWS Glue supports resource-based policies to control access to Data Catalog resources. These resources include databases, tables, connections, and user-defined functions. For more information, see AWS Glue Resource Policies in the AWS Glue Developer Guide.

When using resource-based policies to limit access to AWS Glue from within Amazon EMR, the principal that you specify in the permissions policy must be the role ARN associated with the EC2 instance profile that is specified when a cluster is created. For example, for a resource-based policy attached to a catalog, you can specify the role ARN for the default service role for cluster EC2 instances, EMR_EC2_DefaultRole as the Principal, using the format shown in the following example:

arn:aws:iam::acct-id:role/EMR_EC2_DefaultRole

The acct-id can be different from the AWS Glue account ID. This enables access from EMR clusters in different accounts. You can specify multiple principals, each from a different account.

Considerations when using AWS Glue Data Catalog

Consider the following items when using the AWS Glue Data Catalog as the metastore with Hive: