Prompt experiments

Use the existing helpers in the command reference. Keep helpers and durable round records under scripts/prompt-experiments/, and raw output under tmp/prompt-experiments/. Applying results to production is a separate task; leave production source and prompt tests unchanged.

Resume or start

A round contains many candidate iterations. Start with its current-state entry in REPORT.md and the relevant frozen prompt. Search older reports for the current failure or hypothesis, then read only the matching findings. Load corpus text, failed answer rows, or reference sections when the next decision needs them. Reuse recorded checks and diagnostics while their inputs remain unchanged.

For a new round, read the destinations and round record sections once, inspect the production prompt and question builder, and select verified corpus records with their exposure history. Record the goal, selection, controls, and reused findings. Declare holdout coverage and separation rules now; collect holdouts only after confirmation passes. Existing unseen reservations may remain private.

For an existing shape, measure fresh shipping as the control; for a new shape, start with a concise instruction and explicit labels. Screening uses corpus order and a recorded seeded shuffle, with 1 attempt per target per order. If no development failures remain, collect more verified coverage or finish without a prompt change.

For an audit of a completed round, read verification and retention and inspect its saved evidence; a new inference run is new evidence.

For inference on a configured machine:

Use scripts/prompt-experiments/.env.local. Reuse the live pangu-eval connection, or attach it to the configured PANGU_CDP_URL with playwright-cli -s=pangu-eval attach --cdp="$PANGU_CDP_URL" after loading the settings into the environment. Try the saved connection before inspecting Chrome settings or rediscovering local values; a successful attachment completes connection setup.
Apply the reuse conditions to the recorded checks. Run the missing or invalidated checks and record any missing runtime metadata.
Once those checks pass, run shipping and the first frozen candidate with sweep.mjs; resume an existing round from its recorded next comparison.