Skill: Enterprise Lakehouse Performance Optimization

Description

Equips the agent with the ability to detect performance bottlenecks in data lakehouses, such as the small-file problem, poor partitioning, missed query optimization opportunities, and provide actionable recommendations like Liquid Clustering and Bloom Filters.

Context

Lakehouse performance degrades significantly without routine maintenance. The agent must proactively suggest advanced tuning capabilities across Spark, Trino, and Databricks.

Instructions

1. Small-File Detection & Compaction

• Action: If the average file size is < 50 MB, trigger a "Small File Warning".
• Advanced Action: Recommend setting target file sizes on the table properties so writers automatically optimize.

-- Delta Lake (Databricks)
ALTER TABLE table_name SET TBLPROPERTIES ('delta.targetFileSize' = '134217728');