LLM Map-Reduce for Large Document Analysis

When to Use

• The target document (or collection of documents) is too large for one context window.
• You need global understanding across the entire text (themes, cross-references, overall narrative, contradictions, or high-level insights).
• Common tasks: full-document summarization, theme extraction, key decision logging, comparative analysis, report generation, codebase architecture review.
• Pure RAG retrieval is insufficient because the task requires synthesis and reasoning across many parts of the document.
• You want a scalable, parallelizable, and auditable process that works with models of any context size.

For initial low-level reconnaissance (e.g., identifying structure, extracting specific patterns, or statistics), combine this pattern with traditional scripting tools (grep, awk, Python single-pass processing) as a preparatory step.

Core Map-Reduce Pattern

1. Chunk — Split the document into manageable, semantically coherent pieces.
2. Map — Independently process each chunk (summarize, extract facts, answer sub-questions, or perform targeted analysis). These steps can run in parallel.
3. Reduce — Aggregate all Map outputs into one coherent final result (comprehensive summary, structured report, answers to questions, or synthesized insights).