Writing simulation scenarios

A scenario is a simulated user's instructions (who they are, what they want) plus agent_expectations (what counts as success). A simulation plays the scenario against the real agent, and an LLM judge grades the transcript.

Runs are disposable. The scenario file is what lasts: it gets reviewed in diffs, re-run for years, and it's what catches a prompt edit that breaks something without anyone noticing.

Before writing a file, look up the exact schema and CLI flags with reading-livekit-docs. Field names and commands change, so this skill doesn't restate them. Conceptually, a file is a named group of scenarios. Each scenario has the simulated user's instructions, the pass criteria, and optionally tags for grouping and per-scenario data the agent can read at runtime. If the CLI offers to add stable per-scenario ids to a file, accept and commit them.

Start from a generated baseline

Don't start from an empty file. LiveKit's generator reads the agent's source and produces a decent first pass much faster than you'd write ten scenarios by hand: