Core Data Engineering

Use this skill for day-to-day Python data engineering decisions: dataframe processing, embedded OLAP SQL, Arrow interchange, ETL structure, and resilience patterns.

When to use this skill

Use this skill when the task involves one or more of:

• Polars transformations and lazy query optimization
• DuckDB SQL analytics, MERGE/upsert, or Parquet querying
• PyArrow table/dataset interchange and Parquet scans
• Python ETL pipeline structure (extract/transform/load, logging, retries, idempotency)
• PostgreSQL-to-lake/warehouse ingestion patterns

If the task is primarily about cloud storage auth/connectors, use:

• @data-engineering-storage-remote-access
• @data-engineering-storage-authentication

If the task is primarily about ACID lakehouse table behavior, use:

• @data-engineering-storage-lakehouse