Databricks Core Workflow A: Delta Lake ETL

Overview

Build production Delta Lake ETL pipelines using the medallion architecture (Bronze > Silver > Gold). Uses Auto Loader (cloudFiles) for incremental ingestion, MERGE INTO for upserts, and Delta Live Tables for declarative pipelines.

Prerequisites

• Completed databricks-install-auth setup
• Unity Catalog enabled with catalogs/schemas created
• Access to cloud storage for raw data (S3, ADLS, GCS)