eval-rubric-language-quality-ar
Installation
SKILL.md
Eval Rubric — Language Quality (Arabic)
When to use this
Apply whenever an AI legal output is in Arabic or contains a substantial Arabic component. This rubric is scored independently of legal accuracy — correct law stated in poor Arabic is a quality failure for a MENA legal AI product. Arabic legal writing has strict conventions in professional practice; outputs that deviate from these conventions undermine user trust.
Requires a human Arabic legal reviewer for high-stakes outputs; LLM judges have limited reliability for detecting subtle Arabic legal register issues.
Scoring (0–5)
| Score | Label | Criteria |
|---|---|---|
| 5 | Excellent | Grammatically correct MSA throughout; correct legal terminology (مكافأة نهاية الخدمة, اتفاقية عدم الإفصاح, قانون العمل); appropriate formal register for the document type; no transliterated loanwords where Arabic equivalents exist; RTL formatting correct; legal style appropriate to the jurisdiction (Khaleeji vs Levantine legal conventions where relevant) |
| 4 | Good | Grammatically correct with minor issues; terminology mostly correct; 1–2 instances of unnecessary transliteration or informal register |
| 3 | Acceptable | Substantially correct but notable terminology issues (e.g., using "كونفيدنشيالتي" instead of "سرية"; using dialectal words in a formal contract); or occasional grammatical errors that do not obscure meaning |
| 2 | Poor | Multiple grammatical errors; significant terminology problems; mixed register (formal and informal in the same document); RTL formatting issues |
| 1 | Very poor | Barely intelligible Arabic; heavy transliteration; appears to be machine-translated from English without legal register correction |
| 0 | Fail | Not Arabic (wrong language output for an Arabic-input request), or completely garbled |