Migrate Databricks Workflows to ZenML

This skill translates Databricks Workflows (Lakeflow Jobs) into idiomatic ZenML pipelines. It handles the full migration workflow: analyzing job definitions and notebook code, classifying each pattern, translating what maps cleanly, flagging what needs redesign, and producing a working ZenML project.

How migration works at a high level

Databricks Workflows and ZenML look similar on the surface -- both define a DAG of tasks/steps with dependencies. But the underlying execution models are fundamentally different. Databricks models orchestration as an explicit DAG of task objects in JSON (each with a task_key, a concrete task type like notebook_task or sql_task, and an explicit depends_on list), with substantial runtime configuration co-located in task settings (compute binding, retries, notifications). ZenML models orchestration as Python function calls forming typed artifact edges, with runtime behavior driven by step/pipeline decorators, stack components, and containerization settings.

This means migration involves two distinct challenges:

1. Structural translation: mapping Databricks' JSON-defined DAG + heterogeneous task types into Python-defined ZenML steps and pipelines
2. Semantic translation: handling the differences in data passing (string-substituted task values vs typed artifacts), execution environment (managed Spark clusters vs containerized steps), and platform-coupled features (Unity Catalog, DBFS, dbutils)

The three mapping types

Every Databricks concept falls into one of these categories: