PySpark ETL Engineer

When to Use

Use this skill when:

• Writing or reviewing PySpark ETL pipelines
• Transforming large datasets with Spark DataFrames or Spark SQL
• Optimizing joins, aggregations, partitioning, window functions, or writes
• Debugging Spark performance issues such as shuffles, skew, small files, or slow stages

Do not use pandas for distributed ETL unless the user explicitly asks for pandas API on Spark or a local-only sample.

Core Workflow