Data pipeline authoring

A data pipeline is NOT a flow. A flow is one runnable that orchestrates steps internally. A data pipeline is a set of independent scripts, each deployed on its own, that form a DAG by reading and writing shared storage assets (DuckLake tables, data tables, S3 objects, volumes, resources) and by declaring execution triggers. The pipeline is visualized and edited at /pipeline/<folder>; every node is a normal workspace script that happens to carry pipeline annotations. When the user asks for a "data pipeline" (or to "ingest / transform / materialize" data across steps), build pipeline-annotated scripts — do NOT build a flow.

Default to DuckDB + DuckLake

A pipeline node that produces a table should almost always be a duckdb node that materializes its output into a DuckLake table with -- materialize ducklake://<name>/<table> (in a DuckDB node the annotation uses SQL -- comment syntax; write the body as a bare SELECT and let the runtime do the write). DuckLake is the default lakehouse store for pipelines and is the shape the pipeline editor is built around, so prefer it unless the work specifically calls for something else:

• postgresql / data tables — only for row-level, OLTP-style mutations against an existing Postgres data table (frequent single-row upserts/updates, transactional reads that an app queries live).
• bun / python3 — only for work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node produces tabular data for downstream steps, write it straight into DuckLake (or a data table) from the same script — see "Ingesting from Python / TypeScript" below — rather than inventing a parallel store.

Do not spread a pipeline across postgres, S3, and DuckLake when one DuckLake lake would do; a consistent DuckLake lakehouse is the goal.

Ingesting from Python / TypeScript

A node that fetches data in code (an API, a SaaS export, a scraper) writes the rows directly into their DuckLake table with wmill.ducklake(), or into a data table with wmill.datatable(). DuckLake already stores every table as parquet files in the workspace object storage, so never stage the raw payload first — no intermediate JSON / CSV / parquet file in S3 and no second "load the raw file" node. One ingestion node, one table.

Pass the rows as a query argument (a list of dicts / array of objects is sent as JSON) and let SQL unnest them; declare the column types so the table gets a real schema: