Considerations and limitations for maintenance jobs - Amazon Simple Storage Service

Considerations and limitations for maintenance jobs

Amazon S3 offers maintenance operations to enhance the performance of your S3 tables or table buckets. These options are file compaction, snapshot management, and unreferenced file removal. The following are limitations and consideration for these management options.

Considerations for compaction

The following considerations apply to compaction. For more information about compaction, see S3 Tables maintenance.

  • Compaction is only supported on Apache Parquet file types.

  • Compaction doesn’t support data type: Fixed.

  • Compaction doesn’t support compression types: brotli, lz4.

Considerations for snapshot management

The following considerations apply to snapshot management. For more information about snapshot management, see S3 Tables maintenance.

  • Snapshots will be preserved only when both criteria are satisfied: the minimum number of snapshots to keep and the specified retention period.

  • Snapshot management deletes expired snapshot metadata from Apache Iceberg, preventing time travel queries for expired snapshots and optionally deleting associated data files.

  • Snapshot management does not support retention values you configure as Iceberg table properties in the metadata.json file or through an ALTER TABLE SET TBLPROPERTIES SQL command, including branch or tag-based retention. Snapshot management is disabled when you configure a branch or tag-based retention policy, or configure a retention policy on the metadata.json file that is longer than the values configured through the PutTableMaintenanceConfiguration API. In these cases S3 will not expire or remove snapshots and you will need to manually delete snapshots or remove the properties from your Iceberg table to avoid storage charges.

Considerations for unreferenced file removal

The following considerations apply to unreferenced file removal. For more information about unreferenced file removal, see Amazon S3 table bucket maintenance.

  • Unreferenced file removal deletes data and metadata files that are no longer referenced by Iceberg metadata if their creation time is before the retention period.

Limits for S3 table and table buckets maintenance

Maintenance operation Property Configurable at table bucket level? Configurable at table level? Default value Minimum value
Compaction targetFileSizeMB No Yes 512MB 64MB
Snapshot management minimumSnapshots No Yes 1 1
Snapshot management maximumSnapshotAge No Yes 120 hours 1 hour
Unreferenced file removal ExpireDays Yes No 3 days 1 days
Unreferenced file removal NoncurrentDays Yes No 10 days 1 days