PySpark Structured Streaming

When to Use

Load this skill when the user needs to:

• Build streaming pipelines with PySpark (Kafka → Iceberg, Kafka → Delta, file watching, etc.)
• Configure watermarks and event-time windowing
• Handle late data, deduplication, and stateful aggregations
• Write to Kafka, Delta Lake, Iceberg, or other sinks from a stream
• Debug streaming queries, monitor lag, and tune checkpointing

Core Model