Token-Aware Semantic Chunking Protocol (Boundary-Aligned RAG Slicing)

Overview

When indexing documentation or codebases for vector search and RAG retrieval, naive splitters slice text by fixed character counts (text[i:i+2000]).

Fixed-character chunking causes severe retrieval degradations:

1. Broken Code Blocks: Splits a TypeScript interface or Python function midway through its body, creating unparseable syntax fragments.
2. Mid-Word Token Clipping: Slices words across token boundaries, corrupting embedding vector representations.
3. Embedding Model Ceiling Exceedance: A character count that translates to 8,250 tokens gets silently truncated by an embedding model with an 8,192-token ceiling.

The Token-Aware Semantic Chunking Protocol measures chunk size strictly using the target tokenizer (tiktoken / BPE) and splits text recursively along semantic boundaries (Markdown Headers, AST Function Blocks, Double Newlines).

Fixed-Character Slicing vs. Token-Aware Semantic Chunking