Data Engineering Pro

Expert-level orchestration of scalable data pipelines and storage architecture. Focuses on turning raw data into structured, actionable insights at scale.

Boundary

data-engineering-pro covers Data Ingestion (Kafka, Kinesis), Orchestration (Airflow, Dagster), Processing (Spark, dbt), and Storage (Snowflake, BigQuery, Data Lakes). It does NOT cover machine learning model training (use machine-learning-pro) or advanced analytics (use data-science-pro).

When to use

• Building robust ETL or ELT pipelines for analytical databases.
• Designing a data warehouse or data lake architecture.
• Implementing real-time event streaming architectures with Kafka.
• Optimizing slow data transformation jobs (e.g., rewriting in PySpark or dbt).

Workflow