magic-linguistic-syntax
When to Use
- Selecting a UD treebank for the target language.
- Cross-lingual parser transfer (no labeled treebank for target).
- Building grammatical-knowledge probes (agreement, case, word order).
- Evaluating whether an LLM has learned target-language syntax (vs surface fluency).
- Annotating new UD data (cross-reference magic-linguistic-annotate).
When NOT to use: the target has rich UD coverage and standard parser fine-tune works → use the parser, no advisory needed. For pure annotation methodology → magic-linguistic-annotate.
The Knowledge Engineers Routinely Miss
-
UD treebank coverage is uneven. 100+ treebanks exist but vary 100× in size. Many class 0-2 languages have only PUD-style 1K-sentence test treebanks — useful for cross-lingual eval, NOT for training.
-
Cross-lingual parser transfer source is picked by URIEL distance, NOT treebank size. Closest typological neighbor with adequate data > distant neighbor with massive data. Route via
magic-linguistic-scopeURIEL output. -
Agreement probes (subject-verb, gender, case) detect grammatical knowledge directly. Better than parsing F1 for evaluating LLMs (which don't expose parses). For each phenomenon, construct minimal-pair probes ("the cat sleeps / sleep") and compute model log-likelihood ratio.
-
Trankit (2021) is generally better than stanza for low-resource UD parsing. Stanza wins on speed + coverage. UDify (2019) is older but multilingual-shared.