LLM Observability & Cost Attribution

You cannot debug, optimize, or cost what you cannot see. Traditional APM assumes deterministic, cheap, fast functions. LLM systems break all three assumptions: the same input yields different outputs, a single user request fans out into a dozen model and tool calls, every call costs real money proportional to tokens, and output quality silently drifts as models, prompts, and data change underneath you. Observability for LLM systems is therefore a first-class engineering discipline, not a dashboard you bolt on later — it is the only way to answer "why did this request fail," "which feature is burning our budget," and "did quality regress after the deploy."

How to use this skill: instrument every LLM call as a span inside a request-level trace (§2), attach the metadata tags that make cost and quality attributable (§5), emit the metrics in §3, watch for drift in §4, and debug from traces in §6. Treat the failure modes in §9 as a pre-ship checklist.

1. Why LLM observability is its own discipline

• Non-determinism. Temperature, sampling, and model updates mean the same prompt produces different responses. You cannot reproduce a bug from inputs alone — you need the exact request/response that was captured, including the model version that served it.
• Multi-step agents. A modern request is a tree: planner → retrieval → tool call → sub-agent → synthesis. A failure three levels deep surfaces as a vague top-level error. Without nested traces you see the symptom, never the cause.
• Cost is variable and large. Spend scales with tokens, not requests. A single runaway agent loop or a bloated context window can cost 100× a normal request. You must measure cost per call to control it.
• Quality drifts silently. No exception is thrown when answers get worse. A provider ships a new model snapshot, your RAG corpus shifts, a prompt edit regresses one intent — and nothing alerts unless you measure quality online.

Concretely, what blindness costs you. A support agent starts giving wrong answers after a Tuesday deploy. Without traces: you know error rate is flat (it didn't error, it answered confidently and wrongly), you can't reproduce the complaint (different output every run), you can't tell if it was the prompt edit, a model snapshot rollover, or stale retrieval, and you have no idea the bad path is also the most expensive one because it retries internally. With traces + tags + online evals: the faithfulness eval dropped at 14:02, every failing trace shows an empty retrieval span, the prompt-version stamp points at the deploy, and the cost-by-feature chart already flagged the spike. Same incident, minutes vs. days.

The discipline in one line: capture every call with enough structure (trace shape) and enough metadata (tags) that you can later ask why it failed and what it cost — broken down by the dimensions your business cares about.