Config Online Evaluations

Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.

Prerequisites

• LaunchDarkly account with AgentControl enabled
• API access token with write permissions
• Existing config with variations (use configs-create skill)
• For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+

API Key Detection

1. Check environment variables - LAUNCHDARKLY_API_KEY, LAUNCHDARKLY_API_TOKEN, LD_API_KEY
2. Check MCP config - Claude: ~/.claude/config.json -> mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY
3. Prompt user - Only if detection fails

Core Concepts