Automated Experimentation

Use this skill when you have a goal you want to move through experimentation (e.g., "increase checkout conversion by 15%", "decrease page load time by 20%").

Core principle: experiment first, then a guarded rollout, or a regular rollout if the plan has no guarded rollouts. Always prove a change on a small, fixed slice of traffic via an A/B experiment before rolling it out. Never roll out blind — the rollout exists only to scale a change the experiment has already shown to work.

Execution mode: once the goal is confirmed (Step 1), run Steps 2–8 end-to-end. Async operations (code implementation, release review, deployment, experiment monitoring, rollout monitoring) should be checked periodically, not tight-polled — see the waiting note in each step. Stop and ask the user only for a planned handoff (merging the PR; any code, review, or deploy step you hand over without GitHub) or if a step fails unrecoverably (repeated failed release reviews, deployment failure, or an inconclusive/losing experiment result).

Confirmation: every write tool that takes confirm needs confirm: true, which you send only after the person has approved that write.

Approvals: if the LaunchDarkly environment requires approval for a change (for example, to turn a flag on or off, start or stop an experiment, or start a guarded rollout), create an approval request instead of the direct change. The first time you change an entity (a flag, an experiment) in an environment, call the change tool directly; don't guess beforehand whether the environment requires approval. If the response has requiresApproval: true, tell the user the change was not made because it needs LaunchDarkly approval, ask who should review it, and submit the returned instructions with create-approval-request (or the approval tool that response names). From then on, send every change to that entity in that environment straight to an approval request with the same instructions you would give the direct call, instead of calling the change tool first. Tell the person who must review it. Continue only after get-approval-requests shows it approved, then apply it with apply-approval-request and go to the next step. If it still shows pending, stop and tell the user it is waiting for review; never apply a pending request, and don't keep re-checking in the same turn. If a reviewer rejects it, stop and report.

Blocked tools: an agent space can block MCP tools that it marks destructive. Likely candidates include, but aren't limited to, toggle-flag, start-experiment-iteration, update-rollout, update-targeting-rules, start-guarded-rollout, and stop-experiment-iteration. A blocked call fails without a LaunchDarkly error, or returns nothing, so you can't tell why it failed. If a write fails without a LaunchDarkly error that explains it, or returns nothing, first read the state back (get-flags or get-experiments): an empty response may still have made the change. If it didn't, don't retry: tell the user the tool may be blocked as destructive in this agent space, and hand them the step as below.

When a step can't succeed through you: a blocked tool, or any other write you can't complete (including create-experiment, create-metric, and the approval tools). Hand the step to the user with exact instructions, not a summary. Give the link, what to click, and every value the call would have sent, using the real names and keys from this run: variation names, percentages, metric keys, randomization unit, stage weights and durations, the winning treatment. If the environment requires approval, tell them the UI will ask them to submit the change for review and who should review it. Then wait until they confirm it's done, read the state back to check it, and continue from the next step. For example:

• Turn the flag on or off, or change its default rule or off variation: open https://app.launchdarkly.com/projects/<project-key>/flags/<flag-key>/targeting?env=<environment-key>&selected-env=<environment-key>, set the toggle, the default rule, or the off variation to the exact variation name or percentages, and save.
• Start a guarded rollout: on the same targeting page, set the default rule to a guarded rollout and give the variation to roll out, the randomization unit, each metric key with roll back on regression, and each stage's percentage and monitoring window from Step 7.
• Start or stop the experiment: open the designUrl that create-experiment returned, or https://app.launchdarkly.com/projects/<project-key>/experiments/<experiment-key>/design?env=<environment-key>&selected-env=<environment-key>. When the user stops it, tell them to choose Control as the winner, for the reason Step 7 gives.