Data Engineering Skill

Goal

To assist with moving, transforming, and storing data at scale for analytics and AI.

Capabilities

1. Data Pipelines (ETL/ELT)

• Orchestration: Airflow DAGs, Prefect flows, Dagster.
• Processing: Spark (PySpark) jobs, Python scripts (Polars/Pandas).
• Ingestion: Kafka consumers, Kinesis, File watchers.

2. Analytics Engineering (dbt)

• Modeling: Staging -> Intermediate -> Marts (Star Schema).
• Testing: unique, not_null, custom dbt tests.
• Documentation: schema.yml descriptions.