QA Agent Testing

Design and run reliable evaluation suites for LLM agents, including tool-using, multi-turn, and multi-agent systems.

Default QA Workflow

1. Define the Agent Under Test (AUT): scope, tools, approval boundaries, out-of-scope requests, and safety rules.
2. Build a starter suite from real work:
   • Smoke suite: 5-8 highest-signal checks for PR gates
   • Regression suite: 15-25 tasks from real failures, tickets, or production traces
   • Refusal/security pack: unsafe requests, prompt injection, tool-output poisoning, and exfiltration attempts
3. Define objective graders first: schema checks, golden traces, deterministic mocks, policy oracles, and tool side-effect checks.
4. Add model-based graders only where objective checks are insufficient; calibrate them and log judge versions.
5. Run offline evals with deterministic controls and trace logging.
6. Add optional online evals or canary comparisons for live traffic.
7. Gate changes on one consistent status model and log regressions.

Use the starter templates in assets/ for day-0 setup. The template keeps 10 tasks + 5 refusals as a starter scaffold, not a best-practice cap.