Readability and Lexical Diversity Metrics

Overview

Compute quantitative complexity targets from a text corpus by measuring readability (grade-level estimates from multiple formulas) and lexical diversity (vocabulary richness corrected for text length). The core principle: no single readability formula or diversity index is sufficient -- use multiple measures, report the range, and correct for text length before comparing. Readability formulas estimate the US grade level required to comprehend text; lexical diversity indices estimate how varied the vocabulary is. Both translate directly into measurable LLM prompt constraints (e.g., "target FK grade 12-14, MTLD above 80, average sentence length 18-22 words").

Why multiple formulas: Flesch-Kincaid and Coleman-Liau can diverge by 2-4 grade levels on the same text because they weight different proxies for complexity (syllables vs. characters, sentence length vs. word length). Reporting only one formula hides this disagreement. The consensus range across formulas is the reliable signal.

When to Use

• Measuring the complexity profile of a text corpus before voice replication
• Establishing quantitative readability targets for LLM prompt constraints
• Comparing complexity across documents, authors, or time periods within a corpus
• Computing lexical diversity with length-corrected measures (MTLD, MATTR)
• Setting a complexity ceiling and floor for generated text
• Feeding readability/diversity metrics into an archetype or persona construction pipeline

When NOT to use: