agent-auto-improvement
Installation
SKILL.md
Agent auto-improvement with Braintrust evals
Goal
Use real production traces to improve a specific behavior of an agent or LLM app. The source of truth is traced production data and a remote Braintrust dataset, not hand-written examples or hardcoded local JSON.
The loop is:
- Pick one narrow, scoreable behavior to improve.
- Inspect bad production traces.
- Categorize what is wrong into a failure taxonomy.
- Capture those cases in a remote Braintrust dataset.
- Write scorers that codify the taxonomy.
- Run an offline eval against the dataset.
- Improve the system prompt, tool definitions, or runtime logic.
- Rerun the eval and compare experiments.
- Only after the offline loop is healthy, add online scorers and automations.