RAG systems

Pipeline stages

1. Ingest — parse, clean, preserve structure (headings, tables).
2. Chunk — size/overlap tuned to content type; keep metadata (source, section).
3. Embed — consistent model; version stored with index.
4. Retrieve — hybrid (keyword + vector) when recall matters.
5. Rerank — optional cross-encoder for top-k precision.
6. Generate — cite sources; refuse when context insufficient.

Chunking heuristics