Data Pipeline Architecture Patterns

Pipeline Architectures

ETL vs ELT (CRITICAL)

• ETL: Extract → Transform (staging) → Load
  • Traditional, on-premise data warehouses
  • Pre-aggregation, complex transformations
• ELT: Extract → Load (raw) → Transform (in warehouse)
  • Cloud warehouses (Snowflake, BigQuery)
  • Leverage warehouse compute power

Lambda Architecture

• Batch layer: historical data processing
• Speed layer: real-time stream processing
• Serving layer: merge batch + real-time views
• Complexity: maintain two codebases