Migrating AWS Glue for Spark jobs to AWS Glue version 6.0

Apache Spark upgrade from 3.5.6 in AWS Glue 5.1 to 4.1.1 in AWS Glue 6.0.

Scala upgrade from 2.12.18 to 2.13.17.

Python upgrade from 3.11 to 3.13.

Open Table Formats (OTF) updated to Hudi 1.1.1, Iceberg 1.11.0, and Delta Lake 4.2.0.

Iceberg format version 3 — Extends data types and existing metadata structures to add new capabilities including:

Spark Declarative Pipelines (SDP) — A new declarative framework for defining end-to-end data pipelines using SQL or DataFrame API, with support for streaming tables and materialized views. For more information, see Spark Declarative Pipelines.

Spark Connect for Interactive Sessions — Enables thin-client connectivity to AWS Glue Interactive Sessions through Spark Connect protocol, supporting remote development workflows.

Arrow-native Python UDFs/UDTFs — Improved performance for Python user-defined functions using Apache Arrow columnar format natively.

Customer-managed or service-generated Python virtual environment (--python-virtual-env) — You can build and provide your own Python venv that AWS Glue attaches to Spark drivers and executors at runtime, providing full control over dependency management. When existing jobs are migrated to AWS Glue 6.0, AWS Glue automatically generates this virtual environment if required.

Streaming enhancements — Real-time mode for stateless streaming with millisecond-level latency. For more information, see Enabling real-time mode for streaming jobs.

Connector upgrades — Updated Amazon Redshift, MongoDB, Snowflake, and other connectors. See Appendix C: Connector upgrades for full version details.

ANSI mode is enabled by default in Spark 4.1. Operations that previously returned NULL on overflow (for example, integer arithmetic, cast operations) now throw exceptions. Set spark.sql.ansi.enabled=false to restore previous behavior.

Spark Declarative Pipelines (SDP) is a new feature with limited support for certain SQL constructs. Refer to the Spark Declarative Pipelines documentation for current limitations.

Iceberg v3 tables created in AWS Glue 6.0 cannot be read by Athena SQL (error: Cannot read unsupported version 3). Use Iceberg v2 for cross-engine compatibility with Athena.

Python 3.13 removes several deprecated modules and changes behavior for some standard library functions. Review the Python 3.13 migration guide for compatibility.

AWS SDK for Java 2.x compatibility with --user-jars-first — AWS Glue 6.0 includes AWS SDK for Java 2.x version 2.44.6. If you use the --user-jars-first job parameter with a custom JAR that bundles an older version of the AWS SDK for Java 2.x, your job might fail with a java.lang.NoSuchFieldError or similar error. These failures occur when the SDK bundled in your custom JAR is missing classes, fields, or methods that the AWS Glue runtime depends on. To avoid this issue, ensure that any custom JAR you supply with --user-jars-first uses AWS SDK for Java 2.x version 2.44.6 or later.

EMRFS has been removed. S3A is the only S3 filesystem in AWS Glue 6.0. The com.amazon.ws.emr.hadoop.fs.EmrFileSystem class is no longer available. Jobs using s3:// paths use S3A automatically. If you previously set EMRFS-specific configurations (for example, fs.s3.consistent.*), remove them.

AWS SDK for Java v1 has been removed. Only AWS SDK v2 (2.44.6) is available. Jobs importing com.amazonaws.services.* packages must migrate to software.amazon.awssdk.services.*.

Scala upgraded from 2.12 to 2.13. Custom JARs compiled against Scala 2.12 don't work. Recompile against Scala 2.13.17. Key changes: JavaConversions removed (use CollectionConverters), MutableList removed (use ListBuffer), parallel collections require separate import.

Spark 4.1 API changes:

CreateSession API validation — Additional validation on session parameters for Interactive Sessions. Invalid configurations that were previously silently accepted might now return errors.

getResolvedOptions behavior change — Argument prefix matching is disabled by default (allow_abbrev=False). Use the full argument name, or pass allow_abbrev=True to getResolvedOptions() to restore the old behavior.

In AWS Glue Studio, choose Glue 6.0 - Supports Spark 4.1.1, Scala 2, Python 3 in Glue version.

In the API, choose 6.0 in the GlueVersion parameter in the UpdateJob API operation.

In the console, choose Spark 4.1.1, Python 3 (Glue Version 6.0) or Spark 4.1.1, Scala 2 (Glue Version 6.0) in Glue version.

In AWS Glue Studio, choose Glue 6.0 - Supports Spark 4.1.1, Scala 2, Python 3 in Glue version.

In the API, choose 6.0 in the GlueVersion parameter in the CreateJob API operation.

[Scala] Recompile custom JARs against Scala 2.13.17. Replace JavaConversions with CollectionConverters.

[Python] Update code for Python 3.13 compatibility. Remove usage of deprecated modules (for example, imp, cgi, cgitb).

[Python] Update boto3 references from 1.40 to 1.42.

[Spark SQL] Review queries for ANSI mode impact. Add spark.sql.ansi.enabled=false if needed.

[SDK] Replace any AWS SDK v1 imports (com.amazonaws.*) with SDK v2 (software.amazon.awssdk.*).

[S3] Remove EMRFS-specific configurations. S3A is now the default and only S3 connector.

[Dependencies] Update --extra-jars to versions compiled for Scala 2.13 and Spark 4.1.

S3 filesystem: EMRFS is no longer available. S3A is the sole S3 connector. If you previously set spark.hadoop.fs.s3a.endpoint.region, continue using it. If not set, ensure your VPC endpoint or network configuration allows S3A to resolve the correct region.

Scala binary compatibility: AWS Glue 6.0 uses Scala 2.13. Any --extra-jars compiled with Scala 2.12 fail with NoSuchMethodError or ClassNotFoundException. Recompile all custom Scala/Java JARs.

Spark SQL behavior changes:

Python version: Python 3.13 is the runtime. The --additional-python-modules feature continues to work but is deprecated. Consider migrating to --python-virtual-env for full dependency control.

AWS SDK: Only SDK v2 is available. If your scripts use boto3 (Python), no change is needed — boto3 continues to work. For Scala/Java jobs using the AWS SDK directly, migrate to v2 APIs.

Migration Guide: Spark Core on the Apache Spark website

Migration Guide: SQL, Datasets and DataFrame on the Apache Spark website

Migration Guide: Structured Streaming on the Apache Spark website

Upgrading PySpark on the Apache Spark website

For migration steps related to AWS Glue 5.0 to AWS Glue 5.1, see Migrating from AWS Glue 5.0 to AWS Glue 5.1.

For migration steps related to AWS Glue 4.0 to AWS Glue 5.0, see Migrating from AWS Glue 4.0 to AWS Glue 5.0.

For migration steps related to AWS Glue 3.0 to AWS Glue 5.0, see Migrating from AWS Glue 3.0 to AWS Glue 5.0.

For migration steps related to AWS Glue 2.0 to AWS Glue 5.0, see Migrating from AWS Glue 2.0 to AWS Glue 5.0.

After completing the steps above, finish migrating to AWS Glue 6.0 by following Migrating from AWS Glue 5.1 to AWS Glue 6.0.

Appendix B: JDBC driver upgrades

Appendix C: Connector upgrades

Appendix D: Open table format upgrades

Iceberg upgraded to 1.11.0 with full support for Apache Iceberg version 3 specifications.

VARIANT data type with variant shredding for optimized semi-structured data queries.

Nanosecond-precision timestamps.

Geospatial data types (Geometry and Geography).

Hudi upgraded to 1.1.1.

Continued support for FTA read and write access on AWS Lake Formation registered tables.

Delta Lake upgraded to 4.2.0.

Continued support for FTA read and write access on AWS Lake Formation registered tables.

Iceberg native table encryption keys are not supported.

Iceberg multi-argument transforms are not supported.

The new Iceberg v3 data types are only supported with Spark DataFrames. These features will not work with DynamicFrames.

Visual ETL in AWS Glue Studio does not support the new Iceberg v3 data types. If you want to use these new features with Visual ETL, we recommend migrating your ETL jobs to Amazon SageMaker Unified Studio.

Fine-grained access control (FGAC) is not supported with VARIANT columns.

Iceberg tables created with 'format-version'='3' cannot be read by Athena SQL (error: Cannot read unsupported version 3). Use Iceberg v2 for cross-engine compatibility with Athena.