Test Your Agent with Scenarios

NEVER invent your own agent testing framework. Use @langwatch/scenario (Python: langwatch-scenario) for code-based tests, or the platform MCP tools for no-code scenarios. The Scenario framework provides user simulation, judge-based evaluation, multi-turn conversation testing, and adversarial red teaming out of the box. Do NOT build these capabilities from scratch.

Determine Scope

If the user's request is general ("add scenarios to my project", "test my agent"):

• Read the full codebase to understand the agent's architecture and capabilities
• Study git history to understand what changed and why — focus on agent behavior changes, prompt tweaks, bug fixes. Read commit messages for context.
• Generate comprehensive scenario coverage (happy path, edge cases, error handling)
• For conversational agents, include multi-turn scenarios (using max_turns or scripted scenario.user() / scenario.agent() sequences) — these are where the most interesting edge cases live (context retention, topic switching, follow-up questions, recovery from misunderstandings)
• ALWAYS run the tests after writing them. If they fail, debug and fix them (or the agent code). Delivering tests that haven't been executed is useless.
• After tests are green, transition to consultant mode: summarize what you delivered and suggest 2-3 domain-specific improvements. See Consultant Mode.

If the user's request is specific ("test the refund flow", "add a scenario for SQL injection"):

• Focus on the specific behavior or feature
• Write a targeted scenario test
• If the test fails, investigate and fix the agent code (or ask the user)
• Run the test to verify it passes before reporting done