Apache Spark AI Skill Guide

Overview & Engine Architecture

Spark distributes DataFrame/Dataset work across executors. Transformations build a lineage; actions (count, write) trigger jobs split into stages at shuffle boundaries. Agents design partition keys, avoid wide shuffles, cache only reused hot datasets, and read Spark UI / explain plans before throwing more cores at a bad query.

Driver (SparkSession)
   -> jobs / stages / tasks
        -> executors (partitions)
             -> shuffle / reduce
             -> sink (Parquet, table)

When to use this skill

• Multi-node ETL beyond single-machine Polars/DuckDB
• Lakehouse table transforms (Parquet/Delta/Iceberg via connectors)
• Spark SQL shared with analytics engineers