Lakehouse Table Format Engineering

Overview

Use this skill when the storage layer itself is an architecture decision. It helps agents reason about Iceberg, Delta, Hudi, table evolution, partitioning, compaction, snapshot behavior, and how batch and streaming jobs interact with shared lakehouse tables.

When to Use

• choosing between Iceberg, Delta, and Hudi
• designing lakehouse datasets with update or merge behavior
• changing partition strategy, snapshot retention, or compaction behavior
• integrating Spark, Flink, or warehouse readers with shared table formats
• implementing CDC-style or incremental lakehouse pipelines

Do not use this for unmanaged flat-file datasets that are not intended to behave like governed tables.

Workflow