scanning-experiments-with-replay-vision

Installation
SKILL.md

Scanning experiments with Replay Vision

The job: "I'm running an experiment. Watch the recordings and tell me what's actually happening in each variant."

A Replay Vision scanner is a standing LLM probe over session recordings (see [[creating-replay-vision-scanners]] for the general mechanics). Scoping one to an experiment fixes the classic ways scanners go wrong, all at once: the exposure filter is derived server-side from the experiment_targeting field instead of hand-authored, the prompt is templated from the hypothesis and variants instead of vague, the population is bounded by enrollment, and the experiment's end date gives the scanner a natural end. This skill covers what is experiment-specific; the generic create/size mechanics stay in the parent skill.

The flow: resolve the experiment → set experiment_targeting so the API derives the exposure filter → pick a template → size it → create disabled → preview the prompt on a few real sessions → let the user enable it.

Step 1: Resolve the experiment

experiment-get returns everything needed: feature_flag_key, the linked feature_flag (its filters.multivariate.variants list is the source of truth for variant keys — parameters.feature_flag_variants can be stale), exposure_criteria, resolved_exposure_event, start_date, end_date, and status. If the user didn't identify the experiment, resolve it via [[finding-experiments]] rather than guessing.

You no longer derive the exposure event to build the scan query — the API does that from experiment_targeting (Step 2). You still need the event name for the per-variant readout join at the end; the readout section covers that derivation where it is used.

Guards before doing anything else:

Installs
20
GitHub Stars
84
First Seen
Sep 4, 2026
scanning-experiments-with-replay-vision — posthog/ai-plugin