Apache Beam & Dataflow (Python) Best Practices

Implement Way's architectural patterns and modern (2025+) best practices when building Dataflow Python pipelines.

1. Unified Pipeline Architecture

• Mode-driven routing: --mode streaming vs --mode batch flag conditionally injects I/O connectors (Pub/Sub vs. BigQuery/GCS) and windowing; transform logic is identical across modes
• Layered files: pipeline.py (PTransform wiring) → transforms.py (DoFn impls) → state_machine.py / domain logic (pure Python, zero Beam imports)
• Event-time first: always develop around event time so backfills produce consistent state

Reference: Way's Pipeline Patterns, Community Best Practices

2. Runner v2 + Streaming Engine