Test Your Agent with Scenarios

NEVER invent your own agent testing framework. Use @langwatch/scenario (Python: langwatch-scenario) for code-based tests, or the langwatch CLI for no-code platform scenarios. The Scenario framework provides user simulation, judge-based evaluation, multi-turn conversation testing, and adversarial red teaming out of the box.

Determine Scope

If the user's request is general ("add scenarios", "test my agent"):

• Read the codebase to understand the agent's architecture
• Study git history to understand what changed and why: focus on agent behavior changes, prompt tweaks, bug fixes. Read commit messages for context.
• Generate comprehensive coverage (happy path, edge cases, error handling)
• When the agent is connected to the platform, read the run parameters it declares before you propose anything (langwatch agent get <id> --format json, under parameters). Each one is a lever the team built into the agent: cover it with one scenario per value that changes the expected behavior, and offer a comparison run across an option list (see Read the agent's levers in the Platform Approach)
• For conversational agents, include multi-turn scenarios, because that's where the interesting edge cases live (context retention, topic switching, recovery from misunderstandings)
• ALWAYS run the tests after writing them, and report the result in the same answer. With the Code approach, that means waiting: run pytest or vitest in the foreground with a timeout that covers every simulation, and never hand it to a background task or end the answer with "I will report when it finishes". A code run you did not wait for is not a run: the turn ends, the report never arrives, and the user is left with test files nobody has seen pass. With the Platform approach, a run is scheduled rather than executed in your process, so do not block on --wait in an interactive turn: trigger it, hand over the link, and say what the page will show (see step 5 of the Platform Approach). Report the result: how many scenarios ran, which passed, and what the failures said. Stop after two rounds of fixes and report what still fails, with what you tried. A loop that keeps rerunning the suite spends the time of the user and tells them nothing until it ends. If they fail, first decide which side is wrong. Change the test only when you have evidence that its criteria or its fixture are wrong; otherwise the agent is what needs the fix (see Improving the Agent When a Scenario Fails below). A scenario that goes green because its assertions got weaker has tested nothing.
• After tests are green, transition to consultant mode (see Consultant Mode below) and suggest 2-3 domain-specific improvements.

If the user's request is specific ("test the refund flow"):

• Focus on the specific behavior; write a targeted test; run it.