create-mcp-eval
Installation
SKILL.md
create-mcp-eval
Generate eval tests for MCP servers using @mcpjam/sdk.
This skill guides you through creating eval test files that measure tool-selection accuracy, argument correctness, and multi-turn reasoning for any MCP server. It works with both Jest and Vitest and supports deterministic (mock) and LLM-driven test patterns.
1. Context Gathering
Before generating any code, collect the following from the user:
| Question | Options | Default |
|---|---|---|
| Connection type | stdio (local binary) or http (SSE/Streamable HTTP URL) |
http |
| Test framework | jest, vitest, or none (SDK-only) |
(detect from repo; fall back to vitest) |
| LLM provider | See Supported Providers table below. Format: provider/model |
(must ask user) |
| Save results to MCPJam | none, auto (saves when MCPJAM_API_KEY is set), or reporter (shared EvalRunReporter). Use an MCPJam API key (sk_…) from Settings → API keys; optionally set MCPJAM_PROJECT_ID to file results under a specific project (defaults to the org’s Default project). |
(must ask user) |
| Tool list | Ask user to paste their tool names or an Agent Brief (see Section 8) | — |