Migrating AWS Glue for Spark jobs to AWS Glue version 4.0

It is based on Apache Spark 3.3.0, but includes optimizations in AWS Glue, and Amazon EMR, such as adaptive query runs, vectorized readers, and optimized shuffles and partition coalescing.

Upgraded JDBC drivers for all AWS Glue native sources including MySQL, Microsoft SQL Server, Oracle, PostgreSQL, MongoDB, and upgraded Spark libraries and dependencies brought in by Spark 3.3.0.

Updated with a new Amazon Redshift connector and JDBC driver.

Optimized Amazon S3 access with upgraded EMR File System (EMRFS) and enabled Amazon S3-optimized output committers, by default.

Optimized Data Catalog access with partition indexes, pushdown predicates, partition listing, and an upgraded Hive metastore client.

Integration with Lake Formation for governed catalog tables with cell-level filtering and data lake transactions.

Reduced startup latency to improve overall job completion times and interactivity.

Spark jobs are billed in 1-second increments with a 10x lower minimum billing duration—from a 10-minute minimum to a 1-minute minimum.

Native support for open-data lake frameworks with Apache Hudi, Delta Lake, and Apache Iceberg.

Native support for the Amazon S3-based Cloud Shuffle Storage Plugin (an Apache Spark plugin) to use Amazon S3 for shuffling and elastic storage capacity.

Row-level runtime filtering (SPARK-32268).

ANSI enhancements (SPARK-38860).

Error message improvements (SPARK-38781).

Support complex types for Parquet vectorized reader (SPARK-34863).

Hidden file metadata support for Spark SQL (SPARK-37273).

Provide a profiler for Python/Pandas UDFs (SPARK-37443).

Introduce Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches (SPARK-36533).

More comprehensive Datasource V2 pushdown capabilities (SPARK-38788).

Migrating from log4j 1 to log4j 2 (SPARK-37814).

Breaking changes

In AWS Glue Studio, choose Glue 4.0 - Supports Spark 3.3, Scala 2, Python 3 in Glue version.

In the API, choose 4.0 in the GlueVersion parameter in the UpdateJob API operation.

In the console, choose Spark 3.3, Python 3 (Glue Version 4.0) or Spark 3.3, Scala 2 (Glue Version 3.0) in Glue version.

In AWS Glue Studio, choose Glue 4.0 - Supports Spark 3.3, Scala 2, Python 3 in Glue version.

In the API, choose 4.0 in the GlueVersion parameter in the CreateJob API operation.

Do your job's external Python libraries depend on Python 2.7/3.6?

All existing job parameters and major features that exist in AWS Glue 3.0 will exist in AWS Glue 4.0.

AWS Glue 3.0 uses Amazon EMR-optimized Spark 3.1.1, and AWS Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

Several Spark changes alone might require revision of your scripts to ensure that removed features are not being referenced.

AWS Glue 4.0 also features an update to EMRFS and Hadoop. For the specific version, see Appendix A: Notable dependency upgrades.

The AWS SDK provided in ETL jobs is now upgraded from 1.11 to 1.12.

All Python jobs will be using Python version 3.10. Previously, Python 3.7 was used in AWS Glue 3.0.

As a result, some pymodules brought out-of-the-box by AWS Glue are upgraded.

Log4j has been upgraded to Log4j2.

For migrating certain connectors, see Connector and JDBC driver migration for AWS Glue 4.0.

The AWS Encryption SDK is upgraded from 1.x to 2.x. AWS Glue jobs using AWS Glue security configurations and jobs dependent on the AWS Encryption SDK dependency provided in runtime are affected. See the instructions for AWS Glue job migration.

You can safely upgrade an AWS Glue 2.0/3.0 job to an AWS Glue 4.0 job because AWS Glue 2.0/3.0 already contains the AWS Encryption SDK bridge version.

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

All existing job parameters and major features that exist in AWS Glue 2.0 will exist in AWS Glue 4.0.

The EMRFS S3-optimized committer for writing Parquet data into Amazon S3 is enabled by default since AWS Glue 3.0. However, you can still disable it by setting --enable-s3-parquet-optimized-committer to false.

AWS Glue 2.0 uses open-source Spark 2.4 and AWS Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

The AWS SDK provided in ETL jobs is now upgraded from 1.11 to 1.12.

AWS Glue 4.0 also features an update to EMRFS, updated JDBC drivers, and inclusions of additional optimizations onto Spark itself provided by AWS Glue.

Scala is updated to 2.12 from 2.11, and Scala 2.12 is not backward compatible with Scala 2.11.

Python 3.10 is the default version used for Python scripts, as AWS Glue 2.0 was only using Python 3.7 and 2.7.

Several dependency updates, highlighted in Appendix A: Notable dependency upgrades.

Any extra JAR files supplied in existing AWS Glue 2.0 jobs might bring in conflicting dependencies because there were upgrades in several dependencies in 4.0 from 2.0. You can avoid classpath conflicts in AWS Glue 4.0 with the --user-jars-first AWS Glue job parameter.

AWS Glue 4.0 uses Spark 3.3. Starting with Spark 3.1, there was a change in the behavior of loading/saving of timestamps from/to parquet files. For more details, see Upgrading from Spark SQL 3.0 to 3.1.

We recommend to set the following parameters when reading/writing parquet data that contains timestamp columns. Setting those parameters can resolve the calendar incompatibility issue that occurs during the Spark 2 to Spark 3 upgrade, for both the AWS Glue Dynamic Frame and Spark Data Frame. Use the CORRECTED option to read the datetime value as it is; and the LEGACY option to rebase the datetime values with regard to the calendar difference during reading.

- Key: --conf
- Value: spark.sql.legacy.parquet.int96RebaseModeInRead=[CORRECTED|LEGACY] --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=[CORRECTED|LEGACY] --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=[CORRECTED|LEGACY]

For migrating certain connectors, see Connector and JDBC driver migration for AWS Glue 4.0.

The AWS Encryption SDK is upgraded from 1.x to 2.x. AWS Glue jobs using AWS Glue security configurations and jobs dependent on the AWS Encryption SDK dependency provided in runtime are affected. See these instructions for AWS Glue job migration:

Upgrading from Spark SQL 2.4 to 3.0

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

Changes in Datetime behavior to be expected since Spark 3.0.

AWS Glue 1.0 uses open-source Spark 2.4 and AWS Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

All jobs in AWS Glue 4.0 will be run with significantly improved startup times. Spark jobs will be billed in 1-second increments with a 10x lower minimum billing duration since startup latency will go from 10 minutes maximum to 1 minute maximum.

Logging behavior has changed significantly in AWS Glue 4.0, Spark 3.3.0 has a minimum requirement of Log4j2.

Several dependency updates, highlighted in the appendix.

Scala is also updated to 2.12 from 2.11, and Scala 2.12 is not backward compatible with Scala 2.11.

Python 3.10 is also the default version used for Python scripts, as AWS Glue 0.9 was only using Python 2.

Python 2.7 is not supported with Spark 3.3.0. Any job requesting Python 2 in the job configuration will fail with an IllegalArgumentException.

A new mechanism of installing additional Python modules through pip is available since AWS Glue 2.0. For more information, see Installing additional Python modules with pip in AWS Glue 2.0+.

AWS Glue 4.0 does not run on Apache YARN, so YARN settings do not apply.

AWS Glue 4.0 does not have a Hadoop Distributed File System (HDFS).

Any extra JAR files supplied in existing AWS Glue 1.0 jobs might bring in conflicting dependencies because there were upgrades in several dependencies in 4.0 from 1.0. We enable AWS Glue 4.0 with the --user-jars-first AWS Glue job parameter by default, to avoid this problem.

AWS Glue 4.0 supports auto scaling. Therefore, the ExecutorAllocationManager metric will be available when auto scaling is enabled.

In AWS Glue version 4.0 jobs, you specify the number of workers and worker type, but do not specify a maxCapacity.

AWS Glue 4.0 does not yet support machine learning transforms.

For migrating certain connectors, see Connector and JDBC driver migration for AWS Glue 4.0.

The AWS Encryption SDK is upgraded from 1.x to 2.x. AWS Glue jobs using AWS Glue security configurations and jobs dependent on the AWS Encryption SDK dependency provided in runtime are affected. See these instructions for AWS Glue job migration.

Upgrading from Spark SQL 2.4 to 3.0

Upgrading from Spark SQL 3.0 to 3.1

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

Changes in Datetime behavior to be expected since Spark 3.0.

AWS Glue 0.9 uses open-source Spark 2.2.1 and AWS Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

All jobs in AWS Glue 4.0 will be run with significantly improved startup times. Spark jobs will be billed in 1-second increments with a 10x lower minimum billing duration because startup latency will go from 10 minutes maximum to 1 minute maximum.

Logging behavior has changed significantly since AWS Glue 4.0, Spark 3.3.0 has a minimum requirement of Log4j2 as mentioned here (https://spark.apache.org/docs/latest/core-migration-guide.html#upgrading-from-core-32-to-33).

Several dependency updates, highlighted in the appendix.

Scala is also updated to 2.12 from 2.11, and Scala 2.12 is not backward compatible with Scala 2.11.

Python 3.10 is also the default version used for Python scripts, as AWS Glue 0.9 was only using Python 2.

AWS Glue 4.0 does not run on Apache YARN, so YARN settings do not apply.

AWS Glue 4.0 does not have a Hadoop Distributed File System (HDFS).

Any extra JAR files supplied in existing AWS Glue 0.9 jobs might bring in conflicting dependencies because there were upgrades in several dependencies in 3.0 from 0.9. You can avoid classpath conflicts in AWS Glue 3.0 with the --user-jars-first AWS Glue job parameter.

AWS Glue 4.0 supports auto scaling. Therefore, the ExecutorAllocationManager metric will be available when auto scaling is enabled.

In AWS Glue version 4.0 jobs, you specify the number of workers and worker type, but do not specify a maxCapacity.

AWS Glue 4.0 does not yet support machine learning transforms.

For migrating certain connectors, see Connector and JDBC driver migration for AWS Glue 4.0.

The AWS Encryption SDK is upgraded from 1.x to 2.x. AWS Glue jobs using AWS Glue security configurations and jobs dependent on the AWS Encryption SDK dependency provided in runtime are affected. See these instructions for AWS Glue job migration.

Upgrading from Spark SQL 2.2 to 2.3

Upgrading from Spark SQL 2.3 to 2.4

Upgrading from Spark SQL 2.4 to 3.0

Upgrading from Spark SQL 3.0 to 3.1

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

Changes in Datetime behavior to be expected since Spark 3.0.

Appendix B: JDBC driver upgrades

Appendix C: Connector upgrades

Spark SQL support improvements:

Any Hudi table created before version 0.10.0 without a primaryKey needs to be recreated with a primaryKey field since version 0.10.0.

Several vulnerabilities (CVEs) were addressed.

Java 8 is natively supported.

If the job is using Arrays of Arrays, with the exception of byte arrays, this scenario can be treated as multidimensional arrays.

The current MongoDB connector supports Spark version 3.1 or later and MongoDB version 4.0 or later.

Due to the connector upgrade, a few property names changed. For example, the URI property name changed to connection.uri. For more information on the current options, see the MongoDB Spark Connector blog.

Using MongoDB 4.0 hosted by Amazon DocumentDB has some functional differences. For more information, see these topics:

The "partitioner" option is restricted to ShardedPartitioner, PaginateIntoPartitionsPartitioner, and SinglePartitionPartitioner. It cannot use default SamplePartitioner and PaginateBySizePartitioner for Amazon DocumentDB because the stage operator does not support the MongoDB API. For more information, see Supported MongoDB APIs, Operations, and Data Types.

Delta Lake now supports time travel in SQL to query older data easily. With this update, time travel is now available both in Spark SQL and through the DataFrame API. Support has been added for the current version of TIMESTAMP in SQL.

Spark 3.3 introduces Trigger.AvailableNow for running streaming queries as an equivalent to Trigger.Once for batch queries. This support is also available when using Delta tables as a streaming source.

Support for SHOW COLUMNS to return the list of columns in a table.

Support for DESCRIBE DETAIL in the Scala and Python DeltaTable API. It retrieves detailed information about a Delta table using either the DeltaTable API or Spark SQL.

Support for returning operation metrics from SQL Delete, Merge, and Update commands. Previously these SQL commands returned an empty DataFrame, now they return a DataFrame with useful metrics about the operation performed.

Optimize performance improvements:

Other notable changes:

Added several performance improvements for scan planning and Spark queries.

Added a common REST catalog client that uses change-based commits to resolve commit conflicts on the service side.

AS OF syntax for SQL time travel queries is supported.

Added merge-on-read support for MERGE and UPDATE queries.

Added support to rewrite partitions using Z-order.

Added a spec and implementation for Puffin, a format for large stats and index blobs, like Theta sketches or bloom filters.

Added new interfaces for consuming data incrementally (both append and changelog scans).

Added support for bulk operations and ranged reads to FileIO interfaces.

Added more metadata tables to show delete files in the metadata tree.

The drop table behavior changed. In Iceberg 0.13.1, running DROP TABLE removes the table from the catalog and deletes the table contents as well. In Iceberg 1.0.0, DROP TABLE only removes the table from the catalog. To delete the table contents use DROP TABLE PURGE.

Parquet vectorized reads are enabled by default in Iceberg 1.0.0. If you want to disable vectorized reads, set read.parquet.vectorization.enabled to false.