Lakeflow Spark Declarative Pipelines (SDP)

IMPORTANT: If this is a new pipeline (one does not already exist), see Quick Start. Be sure to use whatever language user has specified only (Python or SQL). Be sure to use Databricks Asset Bundles for new projects.

Critical Rules (always follow)

• MUST confirm language as Python or SQL. Stick with that language unless told otherwise.
• MUST if not modifying an existing pipeline, use Quick Start below.
• MUST create serverless pipelines by default. ** Only use classic clusters if user explicitly requires R language, Spark RDD APIs, or JAR libraries.

Required Steps