Write MCP Host Simulation Evals

Generate LLM host simulation evals for MCP servers using @gleanwork/mcp-server-tester. In mcp_host mode, a real LLM receives a natural language scenario and decides which MCP tools to call — testing tool discoverability, parameter clarity, and description quality.

When to Use mcp_host Mode

Use mcp_host mode when you need to verify:

• Does the LLM know which tool to call for a given task?
• Does the LLM fill in parameters correctly without hints?
• Does the tool description accurately represent what the tool does?

Use direct mode instead for:

• Regression testing known inputs/outputs (faster, free, deterministic)
• Testing tool logic and error handling
• Schema validation

mcp_host mode calls a real LLM API. Each test costs ~$0.003–0.02 depending on the provider and tool count.