Apache Spark Best Practices

Core Principles

• Use DataFrame/Dataset API — never RDD unless absolutely necessary
• Use built-in functions — avoid UDFs (they are black boxes to the optimizer)
• Enable Adaptive Query Execution (AQE) and let it handle partition coalescing and skew
• Use Kryo serialization for all RDD and broadcast operations
• Push filters and column projections as early as possible
• Batch writes into well-sized files (128 MB–1 GB); avoid the small files problem
• Monitor the Spark UI — check shuffle sizes, spill, GC time, and task skew

Application Structure and Configuration (HIGH)

app-spark-session

Use SparkSession.builder as the single entry point.

Set appName meaningfully for identification in the Spark UI. Externalize cluster-specific configuration — never hardcode executor counts, memory, or paths in application code.