Data Engineering Best Practices

Use this skill for production architecture and standards decisions: storage layout, lifecycle, incremental semantics, schema evolution, quality checks, and cost/performance tradeoffs.

When to use this skill

Use for:

• Designing Bronze/Silver/Gold or equivalent data layers
• Choosing append vs overwrite vs merge behavior
• Partitioning and file-size strategy
• Defining schema evolution policy
• Setting testing/observability guardrails
• Establishing retention + cost controls