Creating MCP agent evals for an MCP tool family

Overview

Build a small, calibrated Langfuse eval suite for one tool family (tasks, storage, runs, …), then use its failures to fix the tools. Core principle: evals are designed from user intent, never from tool descriptions — the eval defines what should work; descriptions get fixed afterward to make naive agents pass it.

Commands, item shapes, probe patterns, and sweep queries: reference.md.

The flow

1. Inventory the tools — every tool and every argument group needs at least one case (the coverage matrix at the end proves it).
2. Probe the platform first. Before writing any case that depends on API behavior (required fields, uniqueness rules, limits, error messages), verify it with a throwaway tsx script against the real API. Never write a case on an assumed contract — that's how you get input values the schema rejects.
3. Two datasets, one per CI gate. mcp-server-evals-pr (the --dataset default) holds the kind: "tool-call" items and gates PRs (PR open/reopen, or the validated label); mcp-server-evals-merge holds the kind: "agent" items and runs on push to master. Every item declares metadata.kind ("tool-call", single-turn, only the first tool call is asserted, no judge, nothing executes, or "agent", multi-turn, run to completion and scored by an LLM judge). A kind: "tool-call" case sets expectedTools (required) and, to pin arguments (not just the tool name), expectedArgs — a flat object, every listed key must deep-equal the captured call's same key, unlisted keys ignored; add mcpToolsOnly: true when a case must isolate MCP-vs-MCP tool choice from Claude Code's built-ins. A kind: "agent" case that provokes an error on purpose (a collision, a not-found, requirement discovery) sets metadata.expectedErrors to the tool name(s) allowed to fail on it — the zero-tool-error gate exempts only those, so it never masks an unrelated failure the way a blanket error-tolerant dataset would. Give the item id the dataset-prefixed shape — pr/<tool>/<slug> in the pr dataset, merge/<family>/<slug> in the merge one, where the middle segment is the tool or tool family (search-actors, tasks, web-fetch, …) and <slug> is the rest.
4. Write cases in waves: 2–3 easy (single tool, explicit input) → 1–2 medium (cross-tool chains, run options) → 2–3 hard (vague user language, error recovery, collisions). Run and review each wave before writing the next.
5. Calibrate on the strongest model first (Opus). A failure there is a case defect or a product gap — never a description problem. Only a calibrated suite (strong model 100%) can attribute weaker-model failures to descriptions.
6. Ladder down (Sonnet → Haiku). Passes-on-Opus-fails-on-Haiku = the tool description or output doesn't carry a naive agent. That's the signal you built the suite for.
7. Fix tools via outputs before descriptions. A steering sentence in the tool's response summary/nextStep reaches every agent on every call; description text gets skimmed. Both output nudges that fixed Haiku failures in the original build were response-text changes.

Diagnosing a failed case — in this order