Context Analysis

Analyze plain text documents to understand their semantic structure and token distribution. This skill helps visualize how tokens are distributed across different semantic components of a document.

Workflow

When analyzing a document, follow these steps:

1. Parse the Text

Convert the input text file to JSON format:

./scripts/parse.sh input.txt > parsed.json

2. Count Tokens

Add token counts using tiktoken (GPT-4o encoding):