ai-coding-agents-observability-evals
Installation
SKILL.md
AI Coding Agents Observability And Evals
Use this skill to design or review the feedback loop around a coding-agent runtime: traces, replayable transcripts, eval packs, regression gates, tool-call grading, latency and cost accounting, and production failure triage.
This skill covers how you operate a coding-agent product after the core runtime exists. It does not replace the runtime skills themselves.
Quick Reference
| Question | Read | Outcome |
|---|---|---|
| What should the trace and telemetry model include? | references/trace-and-telemetry-model.md | Durable trace schema, session correlation, event stages, and replay boundaries |
| How should evals, regressions, and cost controls work? | references/evals-regression-and-cost-ops.md | Golden tasks, iterative self-extension packs, trajectory scorecards, and cost-aware release gates |
| How do I use the eval/trace substrate to improve the harness itself? | references/harness-self-evolution.md | Closed-loop harness evolution: three observability pillars, falsifiable-contract edits, attribution |
| How does OpenAI Codex combine rollout replay, SQLite state, doctor reports, and telemetry? | references/openai-codex-rollout-doctor-telemetry.md | Replay artifacts, rebuildable state indexes, redacted diagnostics, W3C traces, token metrics |
| How does Codex wire OTel exporters and what analytics events exist? | references/openai-codex-otel-config.md | Schema lookup step, OTel vs proprietary analytics boundary, tracestate propagation and cardinality rules |