Build Data Pipeline

Use this skill when building or maintaining transactional lakehouse infrastructure: ingestion pipelines, table formats, schema contracts, Write-Audit-Publish (WAP) validation, dead-letter quarantine (DLQ), and orchestration. For one-off exploratory queries, use analyze-data instead.

When to Use

• implementing transactional lakehouse pipelines with Apache Iceberg v3 (metadata-only restructuring, Variant type) or Delta Lake 4.4 UniForm
• enforcing Open Data Contract Standard (ODCS v3.2.0) at producer boundaries before ingestion
• staging writes to isolated snapshot branches (wap_audit_<run_id>) for atomic Write-Audit-Publish (WAP)
• auditing branch data in-process via DuckDB zero-copy Arrow memory before publishing to main
• routing poisoned or non-conforming records into Dead-Letter Queue (DLQ) quarantine with circuit breaker wired to contract SLA fields
• enforcing deterministic upsert MERGE INTO with composite SHA-256 cryptographic natural keys
• configuring S3 object storage prefix hashing (write.object-storage.enabled = true) to prevent 503 throttling
• executing automated table maintenance: 256MB bin-pack compaction, 7-day snapshot expiration, 72h vacuum
• implementing Iceberg REST Catalog protocol: scan planning, ETag-based freshness validation, Idempotency-Key commits, credential vending
• mandating native table encryption (KMS) for all Iceberg and Delta Lake tables
• emitting OpenLineage explicit lineage facets for end-to-end pipeline traceability

Core Rules