magic-linguistic-annotate

When to Use

• Designing a new annotation project (NER, POS, parsing, sentiment, MT eval, anything).
• Selecting an IAA metric for the task.
• Calculating IAA from given counts.
• Running adjudication after multi-annotator pass.
• Active-learning sample selection for limited annotation budget.
• Deciding whether to ship a single-annotator gold dataset (usually NO).

When NOT to use: purely synthetic-data generation → no human labels needed. Adapter / training itself → magic-linguistic-transfer. Existing gold-standard reuse without modification → just use it.

The Knowledge Engineers Routinely Miss

1. Cohen κ is misleading on highly skewed classes. When 90% of items are "negative" and annotators agree on most by chance, κ underestimates real agreement. Use PABAK (Prevalence-Adjusted Bias-Adjusted κ) or report F1-complement.

2. Krippendorff α handles missing data + ordinal scales that κ doesn't. For serious projects with > 2 annotators, partial-coverage data, or ordinal labels (Likert): default α.

3. γ (gamma) is for span/unitized tasks (NER spans, coreference chains, discourse units). Don't use κ for spans — it doesn't model span boundaries.