chunking-strategy

Malicious tool definition detected

Tool: references/implementation.md [1/3] Description: # Complete Implementation Guidelines This document provides comprehensive implementation guidance for building effective chunking systems.

Tool: references/implementation.md [2/3] Description: analysis) # Size appropriateness: Are chunks within optimal size range?

Tool: references/implementation.md [3/3]

Malicious tool definition detected

Tool: references/research.md [1/2] Description: # Key Research Papers and Findings This document summarizes important research papers and findings related to chunking strategies for RAG systems. ## Seminal Papers ### "Reconstructing Context: Evaluating Advanced Chunking Strategies for RAG" (arXiv:2504.19754) **Key Findings**: - Page-level chunking achieved highest average accuracy (0.648) with lowest variance across different query types - Optimal chunk size varies significantly by document type

Tool: references/research.md [2/2] Description: - Contract clause boundaries serve as natural chunk separators - Case law benefits from hierarchical chunking **Best Practices**: - Preserve legal citation structure - Use clause and section boundaries - Maintain context for legal definitions and references ### Financial Documents #### "SEC Filing Chunking for Financial Analysis" **Key Findings**: - Table preservation critical for financial data - XBRL tagging provides natural segmentation - Risk f

Malicious tool definition detected

Tool: references/semantic-methods.md [1/5] Description: # Semantic and Contextual Chunking Methods This document provides comprehensive coverage of semantic and contextual chunking approaches for advanced RAG systems.

Tool: references/semantic-methods.md [2/5] Description: boundary > start_idx: chunk_sentences = sentences[start_idx:boundary + 1] chunk_text = " ".join(chunk_sentences) chunks.append({ "text": chunk_text, "sentence_count": len(chunk_sentences), "start_sentence": start_idx, "end_sentence": boundary, "method": method }) start_idx = boundary + 1 # Add remaining sentences if start_idx < len(sentences): chunk_sentences = sentences[start_idx:] chunk_text = " ".join(chunk_sentences) chunks.append({ "te

Tool: references/semantic-methods.md [3/5] Description: contextualized_chunks.append(contextualized_chunk) return contextualized_chunks def _generate_document_summary(self, text): """Generate a summary of the entire document""" try: prompt = f""" Please provide a brief summary (maximum 100 words) of this document: {text[:1000]}...

Tool: references/semantic-methods.md [4/5] Description: enumerate(zip(chunk_texts, chunk_embeddings)): chunks.append({ "text": chunk_text, "embedding": embedding, "chunk_index": i, "method": "late_chunking" }) return { "chunks": chunks, "chunk_size": best_chunk_size, "method": "late_chunking" } def semantic_late_chunking(self, text, semantic_model=None): """Combine late chunking with semantic boundary detection""" # Generate late chunks late_results = self.late_chunk_embedding(text) best_chunk_s

Tool: references/semantic-methods.md [5/5] Description: finds relevant documents from the knowledge base based on the user's query.

Malicious tool definition detected

Tool: references/strategies.md [1/2] Description: # Detailed Chunking Strategies This document provides comprehensive implementation details for all chunking strategies mentioned in the main skill.

Tool: references/strategies.md [2/2] Description: = f""" Given the following document and a chunk from it, provide a brief context that helps understand the chunk's meaning within the full document.

Malicious tool definition detected

Tool: references/tools.md [1/2] Description: # Recommended Libraries and Frameworks This document provides a comprehensive guide to tools, libraries, and frameworks for implementing chunking strategies.

Tool: references/tools.md [2/2] Description: ## Evaluation and Testing ### RAGAS **Installation**: ```bash pip install ragas ``` **Key Features**: - RAG evaluation metrics - Answer quality assessment - Context relevance measurement - Faithfulness evaluation **Example Usage**: ```python from ragas import evaluate from ragas.metrics import ( faithfulness, answer_relevancy, context_relevancy, context_recall ) from datasets import Dataset # Prepare evaluation data dataset = Dataset.from_dict({ "ques

Malicious tool definition detected

Tool: references/visualization-tools.md [1/4] Description: # Visualization and Evaluation Tools This document covers tools and methodologies for visualizing chunking strategies and evaluating their effectiveness.

Tool: references/visualization-tools.md [2/4] Description: chunks = self._apply_strategy(text, strategy) chunk_sizes_list = [len(chunk['text']) for chunk in chunks] all_chunk_sizes.append(chunk_sizes_list) labels.append(f"Size {size}") # Box plot ax1.boxplot(all_chunk_sizes, labels=labels) ax1.set_title('Chunk Size Distribution') ax1.set_ylabel('Chunk Size (characters)') ax1.grid(True, alpha=0.3) # Histogram for i, (sizes, label) in enumerate(zip(all_chunk_sizes, labels)): ax2.hist(sizes, alpha=

Tool: references/visualization-tools.md [3/4] Description: List, Dict import time class StreamlitChunkingDashboard: def __init__(self): self.chunking_strategies = { "Fixed Size - Small": {"chunk_size": 100, "overlap": 0}, "Fixed Size - Medium": {"chunk_size": 250, "overlap": 0}, "Fixed Size - Large": {"chunk_size": 500, "overlap": 0}, "With Overlap - 10%": {"chunk_size": 250, "overlap": 10}, "With Overlap - 25%": {"chunk_size": 250, "overlap": 25}, "Semantic": {"method": "semantic", "threshold":

Tool: references/visualization-tools.md [4/4]

Malicious tool definition detected

Tool: SKILL.md Description: --- name: chunking-strategy description: Provides optimal chunking strategies in RAG systems and document processing pipelines. Use when building retrieval-augmented generation systems, vector databases, or processing large documents that require breaking into semantically meaningful segments for embeddings and search.

Malicious tool definition detected

Tool: references/advanced-strategies.md [1/4] Description: # Advanced Chunking Strategies This document provides detailed implementations of 11 advanced chunking strategies for comprehensive RAG systems.

Tool: references/advanced-strategies.md [2/4] Description: tokens""" import tiktoken encoding = tiktoken.get_encoding(encoding_name) tokens = encoding.encode(text) chunks = [] for start in range(0, len(tokens), self.step_size): end = min(start + self.window_size, len(tokens)) window_tokens = tokens[start:end] chunk_text = encoding.decode(window_tokens) chunks.append({ "text": chunk_text, "start_token": start, "end_token": end - 1, "token_count": len(window_tokens), "overlap": self.window_size -

Tool: references/advanced-strategies.md [3/4] Description: < len(embeddings) - 1: similarity = cosine_similarity( embeddings[j].reshape(1, -1), embeddings[j + 1].reshape(1, -1) )[0][0] local_similarities.append(similarity) # Use local average for comparison if local_similarities: local_avg = np.mean(local_similarities) current_similarity = local_similarities[-1] # Create boundary if current similarity is significantly lower than local average if current_similarity < local_avg * threshold: bounda

Tool: references/advanced-strategies.md [4/4] Description: # Find sentences before and after chunk_start = full_text.find(chunk) chunk_end = chunk_start + len(chunk) # Get preceding and following context pre_context = full_text[max(0, chunk_start - 200):chunk_start] post_context = full_text[chunk_end:chunk_end + 200] context_parts = [] if pre_context.strip(): context_parts.append(f"Preceding: {pre_context.strip()}") if post_context.strip(): context_parts.append(f"Following: {post_context.strip()

Malicious tool definition detected

Tool: references/evaluation.md [1/3] Description: # Performance Evaluation Framework This document provides comprehensive methodologies for evaluating chunking strategy performance and effectiveness.

Tool: references/evaluation.md [2/3]

Tool: references/evaluation.md [3/3]