Pyflow API Spec

Routing lives elsewhere. Whether a request should be authored in pyflow or the DataStage SDK, and whether it is a create or an edit, is decided by the di-agent-flow-lifecycle skill (AUTHOR state). This file is the pyflow language reference and the mechanics of using it. Before working with flows, you must load the di-agent-flow-lifecycle skill. Do not write any Pyflow code before loading the lifecycle skills.

Usage Guidance

Pyflow is declarative intent; the pyflow compiler lowers the DSL to an engine-specific imperative flow, producing an accurate functional plan for what you declare. You express the user's goal; the compiler picks the stages and the wiring. That is why pyflow needs no stage-level DataStage expertise to use.

The compiler validates your flow and gives detailed compile-time feedback before any asset is published, guarantees correctness, and sets up connection metadata for you — all at a fraction of the tokens of the SDK. The result is a pyflow-native flow that round-trips cleanly for later edits.

Pyflow's value is building structure — sources, joins, filters, and the wiring between them — the expensive, error-prone part to hand-author in the SDK. Expressions and stage properties are cheap to add in the SDK once the structure exists.

Do not abandon pyflow because one function or stage isn't supported. A flow that needs, say, a regex extraction pyflow lacks still starts in pyflow: bootstrap the sources, filter, and join, then splice the missing piece into the generated SDK. Hand-authoring a new join or source from scratch is the SDK's least reliable path — never do it when pyflow can scaffold it.

Creating vs. overwriting a flow

create_pyflow has two modes, selected by replace_flow_id: