Considerations and limitations for maintenance jobs
Amazon S3 offers maintenance operations to enhance the performance of your S3 tables or table buckets. These options are file compaction, snapshot management, and unreferenced file removal. The following are limitations and consideration for these management options.
Topics
Considerations for compaction
The following considerations apply to compaction. For more information about compaction, see S3 Tables maintenance.
-
Compaction is only supported on Apache Parquet file types.
-
Compaction doesn’t support data type: Fixed.
-
Compaction doesn’t support compression types:
brotli
,lz4
.
Considerations for snapshot management
The following considerations apply to snapshot management. For more information about snapshot management, see S3 Tables maintenance.
-
Snapshots will be preserved only when both criteria are satisfied: the minimum number of snapshots to keep and the specified retention period.
-
Snapshot management deletes expired snapshot metadata from Apache Iceberg, preventing time travel queries for expired snapshots and optionally deleting associated data files.
-
Snapshot management does not support retention values you configure as Iceberg table properties in the
metadata.json
file or through anALTER TABLE SET TBLPROPERTIES
SQL command, including branch or tag-based retention. Snapshot management is disabled when you configure a branch or tag-based retention policy, or configure a retention policy on themetadata.json
file that is longer than the values configured through thePutTableMaintenanceConfiguration
API. In these cases S3 will not expire or remove snapshots and you will need to manually delete snapshots or remove the properties from your Iceberg table to avoid storage charges.
Considerations for unreferenced file removal
The following considerations apply to unreferenced file removal. For more information about unreferenced file removal, see Amazon S3 table bucket maintenance.
-
Unreferenced file removal deletes data and metadata files that are no longer referenced by Iceberg metadata if their creation time is before the retention period.
Limits for S3 table and table buckets maintenance
Maintenance operation | Property | Configurable at table bucket level? | Configurable at table level? | Default value | Minimum value |
---|---|---|---|---|---|
Compaction | targetFileSizeMB | No | Yes | 512MB | 64MB |
Snapshot management | minimumSnapshots | No | Yes | 1 | 1 |
Snapshot management | maximumSnapshotAge | No | Yes | 120 hours | 1 hour |
Unreferenced file removal | ExpireDays | Yes | No | 3 days | 1 days |
Unreferenced file removal | NoncurrentDays | Yes | No | 10 days | 1 days |