Spark

Purpose

Write Spark jobs whose cost is understood. Almost all Spark performance problems are one of three things: too much shuffle, skewed partitions, or reading far more data than the query needs.

When to Use

• Building or reviewing a Spark pipeline.
• A job that is slow, failing with out-of-memory errors, or has one straggling task.
• Tuning partitioning and join strategy.
• Reading the Spark UI to diagnose a stage.

Capabilities