SDP/DLT Expectations Patterns

Naming: Databricks rebranded DLT to Spark Declarative Pipelines (SDP). The modern Python API is from pyspark import pipelines as dp with @dp.table() decorators. However, expectations decorators (@dlt.expect_all_or_drop(), @dlt.expect_all()) remain in the legacy import dlt API. This skill uses the legacy API until expectations are migrated to dp.

Overview

All Silver layer tables use data quality expectations loaded from a Unity Catalog Delta table. This skill standardizes the Delta table-based approach for portable, maintainable, and runtime-updateable data quality management.

Key Patterns:

1. Delta Table for Rules Storage - Single source of truth in Unity Catalog
2. Rules Loader Module - Pure Python functions to load rules at runtime
3. @dlt.expect_all_or_drop() Decorator - Strict enforcement pattern
4. Direct Publishing Mode - Fully qualified table names with get_source_table() helper
5. Severity-Based Filtering - Critical vs warning rules

Official Reference: Portable and Reusable Expectations

When to Use This Skill