databricks-ml-training
ML Training on Databricks
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Train with MLflow → register to Unity Catalog → consume the same artifact as either a batch Spark UDF over Delta or (when low-latency is required) a real-time serving endpoint.
Always train on Databricks (serverless job or notebook), never in the local Python process the agent is running in. Local training has no access to the silver tables, no MLflow tracking server, no UC registry path, and dies if the chat session drops — submit
databricks jobs submit --no-wait(see "Train + deploy as a serverless job" below). Only fall back to local execution if the user explicitly asks for it.
If you need to deploy a real time model serving endpoint after the model is registered (creating endpoints, traffic config, version-swapping, querying, Foundation Model API endpoints), see databricks-model-serving.
| Consumption | When | How |
|---|---|---|
| Batch UDF | Dashboards, daily/hourly scores, predictions read by Genie/Dashboards or an app (often synced to a Lakebase table) | mlflow.pyfunc.spark_udf(...) → INSERT INTO gold_predictions |
| Real-time endpoint | Score on a user action (fraud at authorization, rec at page load) — sub-100ms | mlflow.deployments.get_deploy_client() (classical) / agents.deploy() (agents). Endpoint lifecycle: see databricks-model-serving. |