ai-coding-agents-observability-evals

Installation
SKILL.md

AI Coding Agents Observability And Evals

Use this skill to design or review the feedback loop around a coding-agent runtime: traces, replayable transcripts, eval packs, regression gates, tool-call grading, latency and cost accounting, and production failure triage.

This skill covers how you operate a coding-agent product after the core runtime exists. It does not replace the runtime skills themselves.

Quick Reference

Question Read Outcome
What should the trace and telemetry model include? references/trace-and-telemetry-model.md Durable trace schema, session correlation, event stages, and replay boundaries
How should evals, regressions, and cost controls work? references/evals-regression-and-cost-ops.md Golden tasks, iterative self-extension packs, trajectory scorecards, and cost-aware release gates
How do I use the eval/trace substrate to improve the harness itself? references/harness-self-evolution.md Closed-loop harness evolution: three observability pillars, falsifiable-contract edits, attribution
How does OpenAI Codex combine rollout replay, SQLite state, doctor reports, and telemetry? references/openai-codex-rollout-doctor-telemetry.md Replay artifacts, rebuildable state indexes, redacted diagnostics, W3C traces, token metrics
How does Codex wire OTel exporters and what analytics events exist? references/openai-codex-otel-config.md Schema lookup step, OTel vs proprietary analytics boundary, tracestate propagation and cardinality rules

When To Use

Installs
19
GitHub Stars
102
First Seen
Aug 12, 2026