Apache Iceberg Best Practices

Core Principles

• Use hidden partitioning with transforms — never create explicit partition columns
• Run table maintenance regularly — expire snapshots, compact files, rewrite manifests
• Use Parquet with ZSTD compression for all data files
• Configure sort order on frequently filtered columns
• Use a production catalog (REST, Glue, or Hive Metastore) — never Hadoop catalog
• Set state TTL / snapshot retention to prevent unbounded metadata growth
• Design for schema evolution from day one — Iceberg tracks columns by ID, not name

Table Design and Schema Evolution (CRITICAL)

schema-evolution-safety

Iceberg schema evolution is safe — use it instead of recreating tables.

Every column gets a unique ID. Renaming, reordering, adding, or dropping columns are metadata-only operations that never rewrite data files.