AI Data Engineering RAG Pipeline

Skill by ara.so — Data Skills collection.

This skill helps you build production-grade local RAG (Retrieval-Augmented Generation) pipelines using the architectural patterns and implementations from Nahid Mahmud's AI & Data Engineering Roadmap. The project focuses on baseline search engines, hierarchical chunking strategies, and rigorous evaluation methodologies.

What This Project Provides

• Day 01: Retrieval contract design, governed corpus creation, golden dataset (questions.jsonl), and evaluation frameworks
• Day 02: BM25 baseline search engine with inverted index, Okapi BM25 ranking, and Recall@10 evaluation
• Day 03: RAG pipeline with hierarchical chunking (document, section, paragraph levels), deterministic chunk IDs, parent-child metadata linkage, and failure mode analysis

Installation & Setup

Clone the repository: