CinéWorld Real-Time Data Engineering Pipeline Skill

Skill by ara.so — Data Skills collection.

Overview

This project implements an end-to-end real-time data engineering pipeline using Apache Kafka for event streaming, PostgreSQL for data warehousing with Medallion Architecture (Bronze/Silver/Gold layers), Apache Airflow for ELT orchestration, and Streamlit for live visualization. Perfect for learning how to build production-grade streaming data pipelines that process 1M+ events.

Installation

Prerequisites

• Docker and Docker Compose
• Python 3.8+
• Virtual environment (recommended)

Setup Steps