Set up an online experiment on an LLM application

An online experiment splits live traffic between a control and a treatment version of an LLM application. A Datadog feature flag assigns each subject to a variant, and an Agent Observability evaluation score (plus cost and token usage) measures the outcome. This skill walks through the whole setup, from the flag to starting the experiment.

Online experiments for Agent Observability may require the feature to be enabled for the user's organization. If any step reports the feature is unavailable, tell the user to contact their Datadog representative rather than retrying.

When to use this skill

• The user wants to test a change (prompt, model, retrieval strategy, tool set) on real traffic rather than a fixed dataset.
• The user has a change on a branch and wants to know whether it is better, cheaper, or both in production.
• The user wants an experiment whose metrics come from Agent Observability evaluations, token counts, or cost.

Do not use it for offline experiments over a dataset (use the LLM Observability experiment tooling), for analyzing an experiment that already ran (agent-observability-experiment-analyzer or the Datadog experiments tooling), or for promoting an existing offline experiment.

Ground rules