BAT - Bot Acceptance Testing

Bot acceptance testing validates that MCP tools work correctly from a real AI agent's perspective. You design test scenarios dynamically, run them via tests/uat/run_uat.py, and evaluate results.

When to Use BAT

• PR validation: Test that tool changes work correctly from an agent's perspective
• Regression detection: Compare behavior between branches
• Integration verification: Ensure MCP tools work end-to-end with real agent CLIs

Workflow

1. Analyze the change: Read the diff, identify which tools are affected
2. Design scenario: Generate a scenario JSON with setup/test/teardown prompts
3. Run the script: Pipe the scenario to python tests/uat/run_uat.py
4. Evaluate summary: Check all_passed per agent. If true, you're done.
5. Dig deeper on failure: Read results_file for full output, stderr, raw JSON
6. Regression check: If test fails, re-run with --branch master to compare