Chunking & Embeddings

Text splitting, ONNX/static embedding generation, RAG pipeline integration

Locations: crates/xberg/src/chunking/ and crates/xberg/src/embeddings/ (both directories, not single files).

Chunking

ExtractionConfig.chunking: Option<ChunkingConfig> drives it. The standalone entry points are chunking::chunk_text(text, &ChunkingConfig, page_boundaries) -> Result<ChunkingResult> (chunking/core.rs) and chunking::rag::chunk_for_rag(text, &ChunkingConfig) (chunking/rag.rs), which upgrades ChunkerType::Text to Markdown and fills each chunk's heading_path.

ChunkingResult { chunks: Vec<Chunk>, chunk_count: usize }. Chunk carries content, chunk_type, metadata, and the optional vectors embedding, sparse_embedding, late_interaction (types/extraction.rs).