Working with Apache Iceberg V3

Apache Iceberg Version 3 (V3) is the latest version of the Apache Iceberg table format specification, introducing advanced capabilities for building petabyte-scale data lakes with improved performance and reduced operational overhead. V3 addresses common performance bottlenecks encountered with Version 2 (V2), particularly around batch updates and compliance deletes.

AWS provides support for the capabilities defined in the Apache Iceberg Version 3 (V3) specification. These capabilities include deletion vectors, row lineage, and column default values. They also include the following data types:

S3 Tables support all of the data types that V3 introduces.

You can use Apache Iceberg V3 with Apache Spark on Amazon EMR, AWS Glue ETL, Amazon SageMaker Unified Studio Notebooks, and tables in AWS Glue Data Catalog, including Amazon S3 Tables.

Key Features in V3

Version Compatibility

V3 maintains backward compatibility with V2 tables. AWS services support both V2 and V3 tables simultaneously, allowing you to:

Getting Started with V3

Prerequisites

Before working with V3 tables, ensure you have:

Creating V3 Tables

Creating New V3 Tables

To create a new Iceberg V3 table, set the format-version table property to 3.

Using Spark SQL:

CREATE TABLE IF NOT EXISTS myns.orders_v3 (  
    order_id bigint,  
    customer_id string,  
    order_date date,  
    total_amount decimal(10,2),  
    status string,  
    created_at timestamp  
)  
USING iceberg  
TBLPROPERTIES (  
    'format-version' = '3'  
)
                

Upgrading V2 Tables to V3

You can upgrade existing V2 tables to V3 atomically without rewriting data.

Using Spark SQL:

ALTER TABLE myns.existing_table  
SET TBLPROPERTIES ('format-version' = '3')
                

What happens during upgrade:

Enabling Deletion Vectors

To take advantage of Deletion Vectors for updates, deletes, and merges, configure your write mode.

Using Spark SQL:

ALTER TABLE myns.orders_v3  
SET TBLPROPERTIES ('format-version' = '3',  
                   'write.delete.mode' = 'merge-on-read',  
                   'write.update.mode' = 'merge-on-read',  
                   'write.merge.mode' = 'merge-on-read'  
                  )
            

These settings ensure that update, delete, and merge operations create Deletion Vector files instead of rewriting entire data files.

Leveraging Row-lineage for Change Tracking

V3 automatically adds row-lineage metadata fields to track changes.

Using Spark SQL:

# Query with parameter value provided  
last_processed_sequence = 47  
  
SELECT   
    id,  
    data,  
    _row_id,  
    _last_updated_sequence_number  
FROM myns.orders_v3  
WHERE _last_updated_sequence_number > :last_processed_sequence
            

The _row_id field uniquely identifies each row, while _last_updated_sequence_number tracks when the row was last modified. Use these fields to:

Using the variant data type

With the variant data type, you can write semi-structured data like JSON directly in your Iceberg tables without defining a fixed schema in advance. You can write data faster while still getting efficient analytical query performance. Iceberg V3 compatible engines shred your semi-structured data into hidden columns as you write it. These hidden columns generate Parquet column statistics that query engines use for optimizations like file pruning.

Creating a table with a variant column using Spark SQL:

CREATE TABLE IF NOT EXISTS myns.events (
    event_id bigint,
    event_timestamp timestamp,
    source string,
    event_data VARIANT
)
USING iceberg
TBLPROPERTIES (
    'format-version' = '3'
)
            

Inserting semi-structured data into the variant column:

INSERT INTO myns.events VALUES (
    1,
    current_timestamp(),
    'web-app',
    PARSE_JSON('{"user_id": "u-1234", "action": "page_view", "page": "/products", "duration_ms": 350}')
);

INSERT INTO myns.events VALUES (
    2,
    current_timestamp(),
    'mobile-app',
    PARSE_JSON('{"user_id": "u-5678", "action": "purchase", "items": [{"sku": "A100", "qty": 2}], "total": 49.99}')
);
            

With S3 Tables, table maintenance for variant columns, including compaction, runs automatically. Compaction consolidates data from semi-structured sources from small files into larger files that Iceberg engines can read more efficiently, improving query performance over time.

Using the geospatial data types

Creating a table with geospatial columns using Spark SQL:

CREATE TABLE IF NOT EXISTS myns.vehicle_telemetry (
    vehicle_id     string,
    event_time     timestamp,
    position       geometry(4326),
    service_area   geography(4326),
    firmware       string
)
USING iceberg
TBLPROPERTIES ('format-version' = '3')
            

Inserting geospatial values:

INSERT INTO myns.vehicle_telemetry VALUES (
    'v-1024',
    TIMESTAMP '2026-09-18 14:22:31.123456',
    ST_SetSrid(ST_GeomFromWKT('POINT (1 2)'), 4326),
    ST_SetSrid(ST_GeogFromWKT('POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))'), 4326),
    'fw-3.2.1'
)
            

Filtering on location:

SELECT vehicle_id, event_time
FROM myns.vehicle_telemetry
WHERE ST_Intersects(
        position,
        ST_SetSrid(ST_GeomFromWKT('POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))'), 4326)
      )
            

Using nanosecond-precision timestamps

Creating a table with a nanosecond timestamp column using Spark SQL:

CREATE TABLE IF NOT EXISTS myns.trades (
    trade_id     bigint,
    executed_at  timestamptz(9),
    symbol       string,
    price        decimal(18,8)
)
USING iceberg
TBLPROPERTIES ('format-version' = '3')
            

timestamp(9) stores a timestamp without a time zone. timestamptz(9) stores one with a time zone.

Using column default values

Adding a column with a default value using Spark SQL:

ALTER TABLE myns.orders
ADD COLUMN currency string DEFAULT 'USD'
            

Rows written before the currency column was added return USD rather than null, and no data files are rewritten. The value is stored once in the table's schema metadata as the field's initial-default and substituted at read time.

Using the unknown type

Creating a table with an unknown column using Spark SQL:

CREATE TABLE IF NOT EXISTS myns.staging_events (
    event_id   bigint,
    payload    string,
    reserved   unknown
)
USING iceberg
TBLPROPERTIES ('format-version' = '3')
            

An unknown column must be optional, always reads as null, and is not written to data files. You can later evolve it to a concrete type.

Best Practices for V3

When to Use V3

Consider upgrading to or starting with V3 when:

Optimizing Write Performance

Optimizing Read Performance

Migration Strategy

When migrating from V2 to V3:

Compatibility Considerations

The V3 data types are supported only for tables that use the Parquet data file format. They are not supported for tables that use the ORC or Avro data file formats.

For V3 tables, the sort and Z-order compaction strategies don't support the variant, geometry, geography, and nanosecond-precision timestamp data types.

Considerations for compaction

Compaction writes shredded variant Parquet files by default. Older readers that do not support shredding might fail to read compacted files. You can disable shredding by setting the table property write.variant.shredding.enabled=false.

Troubleshooting

Common Issues

Getting Help

Pricing

Availability

Apache Iceberg V3 support for deletion vectors and row lineage is available across all AWS Regions where Amazon EMR, AWS Glue Data Catalog, AWS Glue ETL, and S3 Tables operate.

Column default values and the geometry, geography, unknown, and nanosecond-precision timestamp data types are available in all AWS Regions where S3 Tables are available.

The variant data type in S3 Tables is available in the following AWS Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), and South America (São Paulo).

Additional Resources