magic-linguistic-transfer
When to Use
- Adding a new language to an existing pretrained LLM (Llama-3, Mistral, Qwen, mBART, NLLB, BLOOM, etc.).
- Choosing LoRA rank, alpha, target modules.
- Choosing between continued pretraining vs LoRA vs full fine-tune.
- Picking a tool (Unsloth, LLaMA-Factory, Axolotl, adapters library, PEFT).
- Designing catastrophic-forgetting mitigation.
- Picking adapter stack (MAD-X language + task adapters).
When NOT to use: training English-only with abundant data → standard fine-tune; no transfer-learning specialist needed. Pure tokenizer audit → magic-linguistic-tokenize. Source language selection without adapter context → magic-linguistic-scope.
The Knowledge Engineers Routinely Miss
-
LoRA rank scales with typological distance, not data size. This is the single most common error. r=8 is fine for English-Spanish; r=64+ for English-Inuktitut. URIEL distance is the right input —
magic-linguistic-scopeprovides it. -
Continued pretraining requires ≥ 1B target tokens AND budget. Below 100M target tokens, CP overfits + forgets source. LoRA + vocab extension is the sane path for class 1-3.
-
Unsloth is 2× faster than LLaMA-Factory for single-GPU QLoRA. LLaMA-Factory wins on multi-GPU + complex multilingual sampling. Axolotl is the YAML-config middle ground. Pick based on setup, not popularity.