RAG Data Pipeline

When to Use

Load this skill when the user needs to:

• Design or implement end-to-end RAG ingestion and retrieval pipelines
• Choose or implement a chunking strategy (fixed-size, recursive, semantic, document-structure-aware)
• Embed documents with OpenAI, Cohere, or local sentence-transformers
• Set up or query vector stores: pgvector, Chroma, Qdrant, or Weaviate
• Build incremental embedding refresh pipelines (hash-based change detection, Airflow orchestration)
• Implement hybrid retrieval (dense vector + BM25/keyword + RRF fusion)
• Add cross-encoder re-ranking to improve retrieval precision
• Filter vector search results with metadata predicates
• Monitor retrieval quality (hit rate, MRR, latency) in production

RAG Architecture Overview