Spark And Distributed Processing

Overview

Use this skill for batch workloads that exceed single-node processing and need distributed execution discipline. It helps agents reason about Spark, managed Spark services such as Glue and EMR, partitioning, joins, storage layout, and failure-safe processing.

When to Use

• building or changing Spark jobs
• choosing between Spark, Glue, EMR, or smaller engines
• debugging expensive joins, skew, shuffle, or memory issues
• designing batch pipelines over large files or partitioned tables
• implementing transformations against lakehouse tables such as Iceberg, Delta, or Hudi

Do not use this for lightweight local transforms that fit comfortably in a single process.

Workflow