zomato-ai-data-engineering-pipeline

Skill by ara.so — Data Skills collection.

Complete batch data engineering pipeline that processes food delivery data through a medallion architecture (Bronze → Silver → Gold) using Amazon S3, Snowflake, dbt, Airflow orchestration, and OpenAI-powered AI capabilities (LLM enrichment, RAG, text-to-SQL).

Project Overview

Pipeline Flow:

CSVs → S3 Data Lake → Snowflake RAW (Bronze) → dbt STAGING (Silver) → dbt MARTS (Gold) → AI Layer

Architecture Layers:

• Bronze (RAW): Direct COPY INTO from S3 via storage integration
• Silver (STAGING): dbt views for cleaning, typing, renaming
• Gold (MARTS): Dimensions, incremental facts (MERGE), business aggregates, SCD2 snapshots
• AI: LLM enrichment, RAG chat, text-to-SQL queries