magic-linguistic-corpus
When to Use
- User asks "where do I get data for [language]"; routed at the start of Acquire phase.
- Building a monolingual training corpus from heterogeneous sources.
- Diagnosing model behavior that suggests corpus problems (register-collapse, eval-set memorization, weird domain bias).
- Auditing an existing corpus before training (dedup stats, contamination, register balance).
When NOT to use: for parallel/bitext data → magic-linguistic-bitext. For tokenizer-level audit → magic-linguistic-tokenize. For per-dataset license/ethics → ALWAYS route through magic-linguistic-ethics after the corpus catalog is identified.
The Knowledge Engineers Routinely Miss
-
Language-ID is paragraph-level, not document-level. Multilingual web pages, code-switched documents (Hinglish, Spanglish, Chinglish), Wikipedia articles with multilingual quotes — all break naive document-level lang-ID. GlotLID at paragraph granularity is the floor.
-
CulturaX + MADLAD-400 overlap heavily for many languages. Naïve union inflates dataset size 2-3× without increasing semantic coverage. Always dedup AFTER concatenation, not within each.
-
MinHash threshold for low-resource is 0.9, not 0.8. The standard 0.8 over-merges short texts. Yoruba MasakhaNER + similar small datasets lose 20-30% of valid distinct entries at threshold 0.8.
-
Bible-NLP dominates many low-resource catalogs (often >40% of available text). Bible-only training produces archaic / liturgical register drift in the resulting model. Flag the percentage in every corpus manifest.