magic-linguistic-codeswitch
When to Use
- User-generated text from bilingual / multilingual communities.
- Building a chatbot for code-switching-prevalent communities.
- Diagnosing model failures on Hinglish / Spanglish / etc.
When NOT to use: monolingual data → no code-switching to handle. For language-ID at paragraph granularity → magic-linguistic-corpus.
Stance
Code-switching is the norm, not noise. ~50% of the world's population is multilingual; their conversational text routinely mixes languages. Filtering CS as data-quality issue is a category error: you're filtering the user's natural way of speaking. Treat CS as first-class.
What's worth knowing
-
Matrix Language Frame model (Myers-Scotton 1993) — one language is the matrix (grammatical structure), the other is embedded (insertions). Detection methods exist; useful for tokenizer + corpus stratification.
-
Per-paragraph LID is the floor for CS-prevalent corpora. Document-level LID averages over CS and gets it wrong. (Cross-reference
magic-linguistic-corpus/references/language_id.md.)