dev-prompt-canary
Installation
SKILL.md
gd dev Prompt Canary Validation
Unit tests in src/authoring/dev-prompts.test.ts only verify substring presence in prompt templates—they cannot catch behavioral regressions in what the LLM actually generates.
Whenever you modify src/authoring/dev-prompts.ts, src/authoring/dev.ts, src/grading/generate-grader.ts, or src/grading/template.grader.ts, run this fast empirical canary workflow to verify that generated task.md and grader.ts files obey the rules in project-evals.
1. Incremental Regeneration Mechanics (~4 Minutes Total)
Running gd dev <guide> from scratch takes ~25 minutes per guide because it generates 3 golden solution patches, a zero-passrate.patch, task.md, grader.ts, and runs full evaluations.
Instead, use dev.ts's incremental file-existence checks inside a throwaway worktree:
task.mdOnly (~45s for 8 targets): Keep existing patches and graders untouched. CallextractGuideDescription+buildTargetTaskPrompt+runAgentin a small runner script (or delete onlytargets/<app>/task.mdand rundevGuide(dir, { test: false, targets: [app] })).grader.tsOnly (~2–3 mins per target): Keeptargets/<app>/patches/*-solution.patch,zero-passrate.patch, andtask.mduntouched. Delete onlytargets/<app>/grader.ts, then rundevGuide(dir, { test: false, targets: [app] })to regenerate and calibrategrader.ts.