LLM Observability

Core Philosophy

1. Trace what matters, not everything blindly — Good observability starts with meaningful spans, metadata, and outcomes, not a pile of noisy logs.
2. Evaluation is part of the product loop — Prompts, retrieval, latency, and cost should be measured as continuously as code regressions.
3. Prompt changes need versioning and evidence — Never ship prompt edits without a way to compare behavior, cost, and failure rate.
4. Human feedback and automated scores complement each other — Neither alone is enough for trustworthy LLM systems.
5. Cost, latency, and quality trade off together — A “better” prompt or model is not better if it wrecks budgets or user response time.

Quick-Start: Common Problems

"We can’t debug bad LLM outputs"

1. Trace request → retrieval → prompt → model → post-processing
2. Capture prompt version, model, latency, token usage, and user/session context
3. Log enough artifacts to reproduce failures safely -> See references/tracing-and-spans.md