Book Chunk — Phase 04a: Canonical chunks.jsonl

Purpose

Generate the single source of truth for retrieval — <book-dir>/.ingest/chunks.jsonl. This file feeds pgvector, OpenAI Vector Stores, Gemini File Search, and Claude Files (via /book-rag-push). One JSONL line per chunk; the only thing that varies per provider is the upload format.

The chunker uses the April-2026 SOTA combination:

• Structure-first — split on H2 / H3 headings; never break a paragraph, footnote, code block, or blockquote mid-content.
• Recursive target — 512 tokens per chunk, 64-token overlap (~12.5%).
• Late chunking (Jina, 2024) — embed the full chapter once with a long-context model, then mean-pool token vectors per chunk. Preserves cross-paragraph anaphora (proven +10–12% on documents with pronoun chains).
• Contextual prefix (Anthropic, Sept 2024) — prepend a 50–100 token LLM-generated context that situates the chunk in book + chapter; embed the prefixed text. Cuts retrieval failure ~67% per Anthropic's own benchmarks.

Invocation

$ARGUMENTS: