dev-prompt-canary

Installation
SKILL.md

gd dev Prompt Canary Validation

Unit tests in src/authoring/dev-prompts.test.ts only verify substring presence in prompt templates—they cannot catch behavioral regressions in what the LLM actually generates.

Whenever you modify src/authoring/dev-prompts.ts, src/authoring/dev.ts, src/grading/generate-grader.ts, or src/grading/template.grader.ts, run this fast empirical canary workflow to verify that generated task.md and grader.ts files obey the rules in project-evals.


1. Incremental Regeneration Mechanics (~4 Minutes Total)

Running gd dev <guide> from scratch takes ~25 minutes per guide because it generates 3 golden solution patches, a zero-passrate.patch, task.md, grader.ts, and runs full evaluations.

Instead, use dev.ts's incremental file-existence checks inside a throwaway worktree:

  • task.md Only (~45s for 8 targets): Keep existing patches and graders untouched. Call extractGuideDescription + buildTargetTaskPrompt + runAgent in a small runner script (or delete only targets/<app>/task.md and run devGuide(dir, { test: false, targets: [app] })).
  • grader.ts Only (~2–3 mins per target): Keep targets/<app>/patches/*-solution.patch, zero-passrate.patch, and task.md untouched. Delete only targets/<app>/grader.ts, then run devGuide(dir, { test: false, targets: [app] }) to regenerate and calibrate grader.ts.

2. Execution Workflow

Installs
44
GitHub Stars
9.0K
First Seen
Oct 31, 2026