Skill: Enterprise Lakehouse Data Quality & Schema Drift

Description

Enables the agent to profile data, detect schema evolution/drift over time, and implement enterprise-grade data quality checks using frameworks like Soda Core or Great Expectations.

Context

Lakehouse architectures enforce schema on write. However, schemas evolve over time. Data quality issues like sudden drops in row counts, increased nulls, or delayed data arrivals can silently break downstream analytics.

Instructions

1. Schema Drift Detection

Compare the schema of the earliest available snapshot/commit with the latest snapshot.

• List Added Columns.
• List Removed Columns.
• List Type Changes (e.g., INT to LONG).

For Delta Lake: Use .history() to find schema changes or parse the transaction log. For Iceberg: Use the catalog to compare schemas across table.history().