Silver Layer Setup - Orchestrator Skill

End-to-end workflow for creating production-grade Silver layer pipelines using Spark Declarative Pipelines (SDP, formerly Delta Live Tables/DLT) with Delta table-based data quality rules, quarantine patterns, streaming ingestion, and monitoring views.

Naming: Databricks rebranded DLT to Spark Declarative Pipelines (SDP) / Lakeflow Declarative Pipelines (LDP). The modern Python API is from pyspark import pipelines as dp. However, our DQ rules framework still uses import dlt (legacy API) because @dlt.expect_all_or_drop() decorators are not yet available in the dp API. When Databricks migrates expectations to dp, both this skill and dlt-expectations-patterns will be updated. New projects may use databricks pipelines init to scaffold an SDP Asset Bundle project.

Time Estimate: 3-4 hours for initial setup, 1 hour per additional table

What You'll Create:

1. dq_rules Delta table - Centralized rules repository in Unity Catalog
2. dq_rules_loader.py - Pure Python module to load rules at runtime
3. silver_*.py - SDP/DLT notebooks with expectations loaded from Delta table
4. silver_pipeline.yml - Serverless SDP pipeline configuration
5. DQ monitoring views - Per-table metrics and referential integrity checks

Decision Tree