Word Frequency

Classical, mechanical. No LLM involvement.

Procedure

1. Tokenize: spaCy tokenizer (handles punctuation, contractions) or nltk.word_tokenize. For code-adjacent corpora, preserve underscores/hyphens.
2. Normalize: lowercase. Optionally lemmatize (spaCy) or stem (Porter/Snowball). Lemmatization is usually better for reporting.
3. Filter:
   • Remove stopwords (spaCy / NLTK lists). Allow custom additions (user-domain boilerplate).
   • Remove pure-numeric tokens unless the user wants dates/ids.
   • Length filter (drop tokens <3 chars).
4. Count: collections.Counter. For n-grams, nltk.ngrams or sliding window.
5. Export:
   • word-frequency.csv: term, count, docs_containing, avg_per_doc
   • Top-N summary with percentile cutoffs.
6. Optional enrichments:
   • TF-IDF instead of raw counts (surfaces distinctive words, not just common ones).
   • Per-metadata-slice frequencies (e.g. per month, per category) — chain with trend-analysis.