emnlp-artifact-evaluation
Installation
SKILL.md
EMNLP Artifact Evaluation
Use this for evidence packaging around an EMNLP submission. EMNLP has no separate artifact-badging committee; the artifact is part of the scientific claim, filed under the Responsible NLP checklist and inspected at reviewer discretion. In NLP the artifact surface is unusually broad — data, labels, prompts, outputs, and scoring code are all first-class — and each has its own failure mode.
What NLP reviewers open, in order
Reviewers with thirty minutes and suspicion follow a predictable path:
| Order | Artifact | What they are checking | Cheap failure |
|---|---|---|---|
| 1 | Data sample | Do instances look like the paper's description? | Examples contradict claimed label definitions |
| 2 | Prompt files | Do prompts match the paper's claimed setup? | Prompt contains hints the paper never mentioned |
| 3 | Scoring script | Is the metric computed the standard way? | Custom normalization inflates the headline metric |
| 4 | Annotation guidelines | Could these instructions produce these labels? | Guidelines answer a different question than the task |
| 5 | Output dumps | Are generations as good as the excerpted ones? | Body examples are the best 5 of 500 |