magic-linguistic-morph
When to Use
- Target language is morphologically complex (agglutinative, polysynthetic, templatic, fusional with rich case).
- Tokenizer fertility audit (
magic-linguistic-tokenize) shows morpheme-tokenization gap. - Building gold morphological annotations.
- Augmenting small training data via paradigm completion.
- Choosing between UniMorph paradigms vs ML segmenter vs FST analyzer.
When NOT to use: Latin/Cyrillic fusional language with low morphology (English, Spanish, German default fine-tune is enough). Pure tokenizer audit → magic-linguistic-tokenize.
The Knowledge Engineers Routinely Miss
-
Tokenizer fertility ≠ morphological complexity. Fertility says "BPE produces too many tokens"; morphology tells you WHY (concatenative vs templatic vs fusional vs polysynthetic). Different mechanisms need different fixes.
-
UniMorph covers 100+ languages with gold paradigms — but coverage varies wildly. Turkish/Finnish/Russian deeply covered; Yoruba/Khmer thinly covered; many class 0-1 absent. ALWAYS check before assuming you need self-supervised segmentation.
-
SIGMORPHON 2022/2023 segmenters are SOTA for unsupervised morpheme segmentation. Use them, NOT BPE-as-segmenter. BPE is a compression algorithm; it doesn't respect morpheme boundaries.
-
HFST/foma rule-based FSTs beat ML segmenters when good rules exist — common case for endangered languages where field linguists have built rules. Look in DELAMAN archives.