Data Lake

Purpose

Design and operate ACID-compliant data lakes on object storage using Delta Lake, Apache Iceberg, or Apache Hudi. Manage table formats, optimize storage layout, enforce schema evolution, and implement time-travel and CDC patterns.

Agent Protocol

Trigger

Exact user phrases: "Delta Lake", "Apache Iceberg", "Apache Hudi", "Lakehouse", "ACID on lake", "time travel", "table format", "compaction", "Z-order", "Hilbert curve", "vacuum", "CDC", "merge-on-read", "copy-on-write", "manifest file", "metadata layer", "schema evolution", "optimize table", "incremental query".

Input Context

Before activating, verify:

• Object storage backend (S3, ADLS, GCS, MinIO)
• Table format preference (Delta, Iceberg, Hudi)
• Compute engine (Spark, Flink, Trino, Presto, Hive)
• Write pattern (append-heavy, update-heavy, CDC stream)
• Query pattern (OLAP, incremental, point lookup, full scan)
• Partition strategy (date, categorical, Z-order dimensions)