Data Engineering Architecture (GCP)

Philosophy

SQL-First. If BigQuery can do it in SQL, do it there. Avoid Dataflow/Spark unless the logic requires procedural code, multi-source joins in-flight, or streaming windowing.
Lean Pipeline. Fewer moving parts = fewer failure modes. Native GCP managed services before custom code.
Two options, one verdict. Never propose a single solution silently. Always present Option A (simple/cheap) and Option B (robust/scalable) with explicit trade-offs, then recommend one with justification.

Before proposing any tool or architecture:

Volume/Velocity — MBs or TBs? Batch or stream? Growth rate?
Latency — Real-time (<1 min)? Near-real-time (1-15 min)? T+1 batch?
Replayability — Can we replay from raw? Can tasks restart safely mid-pipeline?

These three answers determine the right tool. Document them before recommending anything.