magic-linguistic-bitext
When to Use
- Building MT data for any low-resource pair.
- Mining parallel sentences from comparable corpora.
- Choosing alignment tool (Vecalign vs hunalign vs Bleualign).
- Generating synthetic bitext via back-translation, dictionary substitution, or pivoting.
- Auditing existing parallel data quality (margin scores, register skew, length filtering).
When NOT to use: for monolingual corpus → magic-linguistic-corpus. For tokenizer fertility on the bitext output → magic-linguistic-tokenize. For ethics/license per source → magic-linguistic-ethics.
The Knowledge Engineers Routinely Miss
-
Vecalign beats hunalign for low-resource. Linear-time + state-of-the-art on Bible-parallel data. Use hunalign only when retrofitting an existing pipeline.
-
Margin threshold 1.06 (NLLB published "clean") over-filters Class 0-1. Use 1.04 + manual spot-check of 50 pairs. The published threshold optimizes for high-resource pairs; low-resource needs gentler filtering or you lose half your usable data.
-
LASER3 has coverage gaps on Bantu and Indigenous Americas. SONAR (Meta 2024) is better for those families. Don't default to LASER3 for everything.
-
Back-translation temperature matters. T=0 produces translationese drift (model parrots English source structure into target); T=0.7-1.0 introduces useful diversity. Always > 0.