Agent Observability Spec Skill

You can't fix what you didn't record. For LLM systems the unit of observability is the trace — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone.

What This Skill Produces

• A trace schema: per-request spans and the fields each must carry
• Metric definitions across health, quality, cost, and behaviour — each with a threshold and owner
• A sampling and retention policy that keeps cost sane and debugging possible
• A privacy note: what logged content contains, who can see it, and how long it lives

Required Inputs

Ask for (if not already provided):

• The system's shape — single LLM call, RAG pipeline, or multi-step tool-using agent
• Traffic volume and cost sensitivity — full tracing at 10M req/day is a budget decision
• What "misbehaving" means here — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
• Existing observability stack (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it