Kitaru replay experiment

Test one candidate condition against known cases and explain whether the available evidence improved, regressed, traded off, or stayed inconclusive. Do not make the deployment decision.

Core contract

• Start from an accepted behavior, exact cohort version, exact evaluator versions and parameters, and one candidate change. Suggest one bounded candidate only when asked.
• Replay starts a fresh agent task from each historical session's stored top-level inputs after applying the override. It does not restore an arbitrary checkpoint, conversation, process memory, adapter instance state, filesystem, or external world state.
• Resolve adapter support and its construction path before asking to run the experiment. A shared replay schema does not prove that an adapter supports a requested override or tool source.
• Require an explicit tool policy for every tool-using run. Omission resolves to live passthrough on the current server and is unsafe as an implicit default.
• Carry exact IDs, versions, evaluator parameters, run-spec evidence, tool policy, failures, and missing results forward.
• Explain remote writes, model and worker compute, cost uncertainty, and possible live effects before execution. One approval after this explanation covers experiment creation and the run start; any tool path with external effects needs separate approval.
• Use established Kitaru product terms only. Do not coin labels for summaries or steps, such as “run card,” “result card,” “agent fingerprint,” or “execution checksum.” Do not replace a Kitaru object with a friendly-sounding alias such as “accepted baseline”; explain the official term when necessary, then use it consistently. In user-facing text, describe what will happen and what the user must decide in ordinary language.
• Prefer native Kitaru MCP operations. Use the structured CLI for built-in waiting or another capability MCP does not expose. Verify installed schemas when they differ from the references.
• Run every Kitaru CLI command and SDK script with KITARU_ACTIVE_SKILL=kitaru-replay-experiment set so the server attributes the resulting activity to this skill.
• Start or restart a user-controlled worker with --concurrency 10. Use KITARU_WORKER_CONCURRENCY=10 only when the launch surface exposes worker settings through environment variables instead of CLI options.
• Never bypass a missing adapter, evidence, comparison, or product contract with direct REST calls or ad hoc local state.

Load references only when needed