magic-linguistic-tokenize

When to Use

• Any new language being added to an existing pretrained model (vocab extension decision).
• Symptoms of high tokenizer fertility (slow generation, context-window blowup, OOV).
• Choosing between training a new tokenizer or extending an existing one.
• Selecting SentencePiece config (BPE vs Unigram, coverage threshold, byte fallback).
• Picking a vocab-extension method (FOCUS / OFA / HyperOfa / full retrain).
• Deciding subword-regularization (BPE-dropout) settings per task.

When NOT to use: the language is well-covered by the existing tokenizer (fertility ≤ 2.0 vs English baseline) AND scope/scripts have already validated the approach. Skip to next phase.

The Knowledge Engineers Routinely Miss

1. Fertility ratio is THE diagnostic. Tokens-per-word vs an English baseline (~1.4 for tiktoken-cl100k_base on Wikipedia English). If your target language hits 3.0+, every downstream cost (latency, context, training compute) is multiplied. Vocab extension is non-negotiable.

2. Class 0-2 languages MUST have byte fallback. Without it, the first OOD test produces <unk> cascades. Every modern tokenizer (SentencePiece 0.1.96+, GPT-2-style byte BPE) supports it — turn it ON.