aiconfig-online-evals
AI Config Online Evaluations
Attach judges to AI Config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.
Prerequisites
- LaunchDarkly account with AI Configs enabled
- API access token with write permissions
- Existing AI Config with variations (use
aiconfig-createskill) - For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.18.0+ or Node.js AI SDK v0.17.0+
API Key Detection
- Check environment variables -
LAUNCHDARKLY_API_KEY,LAUNCHDARKLY_API_TOKEN,LD_API_KEY - Check MCP config - Claude:
~/.claude/config.json->mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY - Prompt user - Only if detection fails
Core Concepts
More from launchdarkly/agent-skills
onboarding
Onboard a project to LaunchDarkly: kickoff roadmap, resumable log, explore repo, MCP, companion flag skills, nested SDK install (detect/plan/apply), first flag. Use when adding LaunchDarkly, setting up or integrating feature flags in a project, SDK integration, or 'onboard me'.
867launchdarkly-flag-discovery
Audit your LaunchDarkly feature flags to understand the landscape, find stale or launched flags, and assess removal readiness. Use when the user asks about flag debt, stale flags, cleanup candidates, flag health, or wants to understand their flag inventory.
815launchdarkly-flag-cleanup
Safely remove a feature flag from code while preserving production behavior. Use when the user wants to remove a flag from code, delete flag references, or create a PR that hardcodes the winning variation after a rollout is complete.
815launchdarkly-flag-create
Create and configure LaunchDarkly feature flags in a way that fits the existing codebase. Use when the user wants to create a new flag, wrap code in a flag, add a feature toggle, or set up an experiment. Guides exploration of existing patterns before creating.
797launchdarkly-flag-targeting
Control LaunchDarkly feature flag targeting including toggling flags on/off, percentage rollouts, targeting rules, individual targets, and copying flag configurations between environments. Use when the user wants to change who sees a flag, roll out to a percentage, add targeting rules, or promote config between environments.
795aiconfig-tools
Give your AI agents capabilities through tools (function calling). Helps you identify what your AI needs to do, create tool definitions, and attach them to AI Config variations.
747