firegoat-onboarding
Draft firegoat evals
npx -y firegoat@latest eval runs the user's own Claude Code, with none of their config and none of their shell's env, in a fresh empty
folder outside the repo (~/.firegoat/runs/<project-hash>/<random>/work, named after nothing in the repo) with its own HOME and one prompt, then runs your checks there. Your job: write 2-3 tests
into firegoat.yaml that tell the maintainer whether an agent can succeed with their
product the way a new user would.
1. Learn the product from the repo (read, don't run)
README*, the docs dir (docs/, *.mdx; quickstarts first), landing copy, package manifests
(package.json, pyproject.toml, go.mod, Cargo.toml) for install names and entry points,
examples/, and an existing firegoat.yaml (keep its tests, see "Updating existing evals" in step 4). Know: the public docs
URL, the credentials it needs (env var names only), and whether the tool needs a fixed port or
starts servers or daemons (step 4).