Chunking

Use this when feeding documents into an LLM context window or a vector store. Xberg chunks two ways: inline during extraction (chunks land on each document's chunks field), or standalone via the chunk command for text you already have. Sizing is character-based by default, or token-based when a tokenizer model is supplied.

Inline during extraction

Turn on chunking with --chunk and the chunks appear on the structured result under chunks: