icse-experiments
Installation
SKILL.md
ICSE Experiments
Design the study to survive an empiricist's audit, because that is who reviews it. ICSE's rigor criterion scores "thoroughness and completeness of an evaluation" (2027 call wording, read 2026-07-08), and the community has strong default expectations per study type that function as unwritten checklists.
Evidence proportional to claim
| Claim shape | Minimum evidence ICSE reviewers expect |
|---|---|
| "Technique X finds more bugs than Y" | Real subject programs, Y actually run (not quoted from its paper), same budget/timeout, statistics + effect size |
| "Developers struggle with Z" | Systematic observation: survey with stated sampling, interviews to saturation, or instrumented behavior — not anecdote |
| "LLM-based tool solves task T" | Ablations over prompts/models, contamination discussion, cost reporting, non-LLM baseline where one exists |
| "Metric M predicts defects" | Multiple projects, time-aware splits, comparison against trivial baselines (size, churn) |
| "Our benchmark/dataset enables research" | Construction protocol, quality validation sample, license clarity, comparison with existing sets |