agent-auto-improvement

Installation
SKILL.md

Agent auto-improvement with Braintrust evals

Goal

Use real production traces to improve a specific behavior of an agent or LLM app. The source of truth is traced production data and a remote Braintrust dataset, not hand-written examples or hardcoded local JSON.

The loop is:

  1. Pick one narrow, scoreable behavior to improve.
  2. Inspect bad production traces.
  3. Categorize what is wrong into a failure taxonomy.
  4. Capture those cases in a remote Braintrust dataset.
  5. Write scorers that codify the taxonomy.
  6. Run an offline eval against the dataset.
  7. Improve the system prompt, tool definitions, or runtime logic.
  8. Rerun the eval and compare experiments.
  9. Only after the offline loop is healthy, add online scorers and automations.

Prerequisites

Installs
12
GitHub Stars
3
First Seen
Jun 29, 2026