project-evals

Installation
SKILL.md

Stage 3: Evaluating guidance for a use case (Needs evals)

This is the third of three stages in creating guidance:

  1. Stage 1: Identifying use cases for a feature
  2. Stage 2: Authoring guidance for a use case
  3. Stage 3: Evaluating guidance for a use case (you are here)

What the eval agent sees vs real-world agents

Real-world coding agents see only guide.md — retrieved via the Skills CLI when a developer asks for help. Every other file in a guide directory is eval infrastructure.

The eval harness runs a coding agent in an isolated workspace initialized with a target base application (daily-grind or devtools-times). The agent receives the prompt from targets/<base_app>/task.md (or legacy Gen1 tasks/task.md) and (in guided mode) access to guide.md via the Skills CLI. The harness then runs targets/<base_app>/grader.ts (or Gen1 grader.ts) against the agent's modified application.

None of the following are ever seen by real-world coding agents:

Installs
48
GitHub Stars
9.0K
First Seen
May 28, 2026