rag-architecture
RAG Architecture
RAG is search + prompt, not magic. The model can only answer from what retrieval puts in the context window, so 80% of RAG quality is the retrieval system and 20% is the prompt. Most "the LLM hallucinated" bugs are actually "retrieval returned garbage and the model dutifully summarized it." Build the pipeline so the right chunks land in context with citations, and tell the model to say "I don't know" when they don't. This skill is the concrete playbook: numbers, methods, tables, and the failure modes that cost precision.
1. The pipeline — eight stages, two phases
Index time (offline, batch): ingest → chunk → embed → index. Query time (online, latency-bound): retrieve → rerank → assemble → generate.
| Stage | Job | Latency budget (typical) |
|---|---|---|
| Ingest | Parse source → clean text + metadata | offline |
| Chunk | Split into retrievable units | offline |
| Embed | Text → vector | offline (docs), ~10–50ms (query) |
| Index | Store vectors + metadata for ANN | offline |
| Retrieve | top-k candidates (dense + sparse) | 10–50ms |
| Rerank | Cross-encoder reorders k→n | 50–300ms |
| Assemble | Select, dedup, order, budget, cite | <5ms |
| Generate | LLM answers from context | 1–10s (dominates) |
Do treat each stage as independently measurable and swappable. Don't optimize generation prompts before you've measured retrieval recall — you're polishing the 20%.