agent-observability-auto-experiment
auto-experiment — local hill-climb improvement loop
This is the local, Claude-Code-driven version of the auto_experiments Temporal/Atlas worker
(domains/ml_observability/apps/apis/auto_experiments/). There, a remote Bits/Code-Gen agent runs
the loop; here YOU (Claude Code) are the agent and run it directly on the current git checkout.
No Temporal, no Code-Gen API — just git commits, a local eval harness, and Datadog LLM-Obs MCP
tools for the data.
Read references/rubrics.md in full before iteration 1 and keep it in mind every iteration.
It holds the non-negotiable rules (never invent a score; what to score; where the data lives; the
harness spec; the metric schema). This file is the control loop; that file is the law.
Security & data handling (read before running)
This skill is local and user-invoked, operating on the user's own checkout with their consent. It has real side effects, so scope them tightly: