magic-linguistic-eval
When to Use
- Reporting quality numbers for any non-English LLM.
- Choosing benchmark + metric for a language pair / task.
- Building grammatical-knowledge probes (BLiMP-style).
- Contamination audit (cross-reference magic-linguistic-corpus).
- Adding fairness eval (per-dialect / per-register breakdown).
When NOT to use: purely qualitative review → human review pipeline. Per-dataset license check → magic-linguistic-ethics. Tokenizer fertility audit → magic-linguistic-tokenize.
Why this is A-tier
The Evaluate phase is the orchestrator's last specialist. Eval results drive release decisions. Picking BLEU over chrF for a morphologically-rich language doesn't just produce one bad number — it produces a misleading number that drives wrong investment decisions for months. Eval is leverage; weak eval cascades.
The Knowledge Engineers Routinely Miss
-
BLEU is pathological for morphologically-rich languages. Single-morpheme edits wreck it as harshly as full mistranslations. For Turkish / Finnish / Swahili / Yoruba / Inuktitut: chrF / chrF++ / spBLEU are primary. Report BLEU as supplementary if at all — never as the headline.
-
COMET coverage varies wildly. COMET-22 has good European + Indic coverage; spotty Bantu / Indigenous Americas. xCOMET-XL extends but still has gaps. ALWAYS check per-language coverage before reporting; a "missing" language gets random numbers.