project-evals
Installation
SKILL.md
Stage 3: Evaluating guidance for a use case (Needs evals)
This is the third of three stages in creating guidance:
- Stage 1: Identifying use cases for a feature
- Stage 2: Authoring guidance for a use case
- Stage 3: Evaluating guidance for a use case (you are here)
What the eval agent sees vs real-world agents
Real-world coding agents see only guide.md — retrieved via the Skills CLI when a developer asks for help. Every other file in a guide directory is eval infrastructure.
The eval harness runs a coding agent in an isolated workspace initialized with a target base application (daily-grind or devtools-times). The agent receives the prompt from targets/<base_app>/task.md (or legacy Gen1 tasks/task.md) and (in guided mode) access to guide.md via the Skills CLI. The harness then runs targets/<base_app>/grader.ts (or Gen1 grader.ts) against the agent's modified application.
None of the following are ever seen by real-world coding agents: