Spark SQL Engineer

When to Use

Use this skill when:

• The user asks for Spark SQL, Hive-compatible SQL, lakehouse SQL, or SQL queries executed by spark.sql
• The task can be expressed more clearly in SQL than PySpark DataFrame code
• You need to review or optimize joins, aggregations, windows, partitions, writes, or query plans
• The data volume is large enough that shuffles, skew, HDFS/file layout, and partition pruning matter

Prefer the pyspark_etl skill for DataFrame-heavy pipeline code. Use this skill for SQL-first answers.

Core Workflow