Zerobus Ingest

Zerobus Ingest (part of Lakeflow Connect) is a managed, push-based ingestion service: a producer sends records and they land in a Unity Catalog managed Delta table, with no broker or ingestion cluster to run. The SDKs do real work for you: OAuth, buffering, offset tracking, durability acknowledgments, and automatic recovery that replays unacknowledged records after transient failures.

FIRST: Use the parent databricks-core skill for CLI install, authentication, and profile selection.

Decide before you build

Recommend Zerobus when data is pushed from an app, service, or device; the destination is a Delta table; the workload is continuous or event-driven and needs a shock absorber (Zerobus lands data in about 5 seconds, and it still fits when minutes are acceptable); and there is one main downstream consumer (or a message bus such as Apache Kafka, Amazon Kinesis, Azure Event Hubs, or Google Cloud Pub/Sub is just a pipe to the lakehouse).

Don't recommend Zerobus when many independent consumers outside the lakehouse need the data (use a message bus such as Apache Kafka, Amazon Kinesis, Azure Event Hubs, or Google Cloud Pub/Sub), or when the data is already in Kafka or another bus and must be pulled into the lakehouse (use the Lakeflow Connect managed Kafka connector or Spark Structured Streaming). Explain why and name the alternative. Before ruling Zerobus out for files, SaaS apps, or databases, check references/choose-an-interface.md: batches of columnar data can go through Arrow Flight, and a SaaS app that can push to a Zerobus-compatible API can use Zerobus.

Prefer an off-the-shelf path. If the source already speaks OpenTelemetry, Kafka, or MQTT, or runs Debezium, Vector, Telegraf, or Cribl, point the user to that path instead of writing SDK code. All of these, plus the full decision tree, are in references/choose-an-interface.md. For existing Kafka producers, read references/kafka.md; for MQTT v5 devices, references/mqtt.md; for columnar batches (Parquet, Arrow, DataFrames), references/arrow-flight.md; for migrating a pipeline from Kafka or a Kafka-compatible system, read references/migration-from-kafka-compatible-systems.md.

Workflow

Treat the request as a conversation: ask one question at a time, use a multiple-choice question tool if one is available, and always offer "Not sure, help me decide". Skip any question the user has already answered.

Many users won't know Databricks. If the user is an app, DevOps, or platform engineer who's new to Databricks, explain each Databricks term in plain language the first time you use it, map it to something they know (a table name is like database.schema.table in Postgres; a service principal is like a service account), and offer to set up what's missing for them. Read references/new-to-databricks.md.