PySpark ETL

This skill covers patterns for building production-grade, testable ETL pipelines with PySpark, Spark SQL, and Apache Iceberg, including project structure, join and window-function idioms, and safe cumulative-table merge patterns.

Workflow for Building a PySpark ETL Job

1. Scaffold the job class — Create a class that manages the SparkSession lifecycle, accepts an injectable session for testing, and exposes an abstract run_job method.
2. Define config via a factory function — Keep config as a plain dataclass; parse CLI args in a separate factory function so tests can construct configs without touching sys.argv.
3. Read source data with a shared, partition-aware reader — Use a generic reader utility for date filters, hour ranges, and latest-partition lookups; keep business filters in the ETL class.
4. Compose the pipeline with .transform() — Chain named methods (read_source().transform(self.enrich).transform(self.merge_with_existing)) so run_job stays pure orchestration.
5. Apply transformations idiomatically — Use select over withColumn chains, explicit join types, explicit window frames, and native functions instead of UDFs.
6. Write with schema-evolution safety — Use .byName() when writing to Iceberg tables so column order doesn't matter.
7. Validate output — Check primary-key uniqueness and null counts on key columns after every write.
8. Test locally — Unit test transformation methods against a local SparkSession with small, hand-built DataFrames.

Project Structure

ETL class scaffold