Eval and Improve

Coding-agent workflow — a /slash-command your coding agent (Claude Code, Codex, others) runs while developing this repo. Invoke it by name (e.g. /eval-and-improve) or describe the task and it triggers automatically.

You're running the agent platform's eval suite, diagnosing every failure, fixing what's in scope, and stopping when all cases pass. The eval wiring lives in evals/cases.py (declares cases) and evals/__main__.py (runner), while fixes may also touch agents/<slug>.py or rare one-line config flips in app/main.py per Step 3. Each case uses agno's built-in AgentAsJudgeEval (LLM judge against a criteria rubric, binary pass/fail) and/or ReliabilityEval (asserts which tools fired) — no custom DSL.

0. Preconditions

• Postgres reachable on 5432: nc -z localhost 5432 returns 0. If not, docker compose up -d agentos-db from the source repo. (docker compose ps is unreliable from worktrees or alternate clones.)
• Venv active: source .venv/bin/activate. If .venv doesn't exist (fresh checkout or worktree), run ./scripts/venv_setup.sh first. evals/cases.py imports the agents directly from agents/, so no AgentOS server has to be running.
• .env populated with OPENAI_API_KEY (and PARALLEL_API_KEY if you have one — the runner pins the expected web-search tool name based on it). evals/__main__.py calls evals.dotenv.load_dotenv() at startup, so you do not need to source .env first. Worktrees don't inherit .env (it's gitignored) — copy it from the source repo if missing.

1. Run the suite

python -m evals               # full suite, concise (response + judge verdicts)
python -m evals -v            # stream the full agent run with rich panels + eval tables
python -m evals --case <name> # single case while iterating