Connecting to data - AWS Glue
Overview of using connectors and connectionsUnified connections

Connecting to data

An AWS Glue connection is a Data Catalog object that stores login credentials, URI strings, virtual private cloud (VPC) information, and more for a particular data store. AWS Glue crawlers, jobs, and development endpoints use connections in order to access certain types of data stores. You can use connections for both sources and targets, and reuse the same connection across multiple crawler or extract, transform, and load (ETL) jobs.

The latest version of the AWS Glue connections schema provides a unified way to manage data connections across AWS services and applications, such as AWS Glue, Amazon Athena, and Amazon SageMaker Unified Studio.

Overview of using connectors and connections

A connection contains the properties that are required to connect to a particular data store. When you create a connection, it is stored in the AWS Glue Data Catalog. You choose a connector, and then create a connection based on that connector.

You can subscribe to connectors for non-natively supported data stores in AWS Marketplace, and then use those connectors when you're creating connections. Developers can also create their own connectors, and you can use them when creating connections.

Note

Connections created using custom or AWS Marketplace connectors in AWS Glue Studio appear in the AWS Glue console with type set to UNKNOWN.

The following steps describe the overall process of using connectors in AWS Glue Studio:

  1. Subscribe to a connector in AWS Marketplace, or develop your own connector and upload it to AWS Glue Studio. For more information, see Adding connectors to AWS Glue Studio.

  2. Review the connector usage information. You can find this information on the Usage tab on the connector product page. For example, if you click the Usage tab on this product page, AWS Glue Connector for Google BigQuery, you can see in the Additional Resources section a link to a blog about using this connector.

  3. Create a connection. You choose which connector to use and provide additional information for the connection, such as login credentials, URI strings, and virtual private cloud (VPC) information. For more information, see Creating connections for connectors.

  4. Create an IAM role for your job. The job assumes the permissions of the IAM role that you specify when you create it. This IAM role must have the necessary permissions to authenticate with, extract data from, and write data to your data stores.

  5. Create an ETL job and configure the data source properties for your ETL job. Provide the connection options and authentication information as instructed by the custom connector provider. For more information, see Authoring jobs with custom connectors.

  6. Customize your ETL job by adding transforms or additional data stores, as described in Starting visual ETL jobs in AWS Glue Studio.

  7. If using a connector for the data target, configure the data target properties for your ETL job. Provide the connection options and authentication information as instructed by the custom connector provider. For more information, see Authoring jobs with custom connectors.

  8. Customize the job run environment by configuring job properties, as described in Modify the job properties.

  9. Run the job.

Unified connections

With Unified connections, you can configure a data connection once, and it can be reused by various services for use cases in data integration, data analytics, and data science. You can create data connections through the AWS Glue console, or custom-built applications using unified data connectivity APIs. With Unified connections, you can set up a connection to a data source using a connection configuration template that is standardized for multiple services. These services (AWS Glue, Amazon SageMaker Unified Studio and Amazon Athena) can share and reuse the same connection with proper permission configuration.

AWS Glue Studio now creates unified connections by default. In the AWS Glue console, you can see the version of the connection in the connections table on the connections page, on the connections detail page, and the connections table in the job details page.

The connection version is visible on Connection details:

Screenshot shows the connections detail on the v2 connection.

The connection version is also visible when viewing all your Connections.

Screenshot shows the connections detail on the v2 connection.

Finally, connection version is visible in the Job details tab for a job.

Screenshot shows the connections detail on the v2 connection.

With version 2 connections, you have the following expanded data connectivity capabilities:

  • Connection type discovery: Support for creating connections using standardized templates. AWS Glue automatically discovers the connection types accessible by you and the required and optional inputs for a given connection type.

  • Reusability: Connection definitions that are reusable across AWS data processing engines and tools like AWS Glue, Amazon Athena, and Amazon SageMaker. Connections now contain AthenaProperties, SparkProperties, PythonProperties which allow to specify compute environment/service specific connection properties in addition to the common properties stored in ConnectionProperties. Athena now creates Connections in AWS Glue by specifying Athena specific properties in the AthenaProperties property map.

  • Data preview: Ability to browse metadata and preview data from connected sources.

  • Connector metadata: Reusable connections may be used in order to discover table metadata.

  • Service linked secrets: Users may provide necessary OAuth, basic or custom authentication credentials in the CreateConnection request. The CreateConnection API creates a Service Linked Secret in your account and stores the credentials on your behalf.