indexion segment

Split text into contextual segments using divergence-based, TF-IDF, or punctuation strategies.

When to Use

• User needs to chunk text for RAG or embedding pipelines
• User wants to split a document into meaningful sections
• User asks to segment text for processing
• Preparing text for similarity analysis at sub-document level

Usage

# Default window divergence strategy
indexion segment <input-file> <output-dir>

# TF-IDF based segmentation
indexion segment --strategy=tfidf <input-file> <output-dir>