Create MCP Eval

Instructions

Every eval is made up of an eval yaml that specifies which agent file to use, which mcp config file to use, which tasks to run, and which MCP-specific assertions to check (if any).

Creating evals from scratch

When creating a full eval from scratch, you will need to:

1. Create one or more tasks, see tasks.md
2. Create a mcp config file, see mcpConfig.md
3. Create an agent file, see agent.md
4. Create a top-level eval file that references the rest of the files, see eval.yaml

However, in most cases you will not be creating an entirely new set of evals from scratch - you will just be modifying or extending an existing eval. In this case, you will only need to modify some of these files.