Phase 7: Observe

Input: a deployed agent. Output: instrumentation, alerts, and a working path from production failure back into the eval set.

Standard monitoring will tell you the agent is up, fast, and error-free while it is confidently wrong on every third request. Latency and error rate are necessary and nowhere near sufficient. You have to measure whether the answers are any good, and that requires deliberate work.

1. Trace every run completely

The trace is the unit of debugging. One span per run, with child spans for each model call and each tool call:

• Model calls: model version, prompt version, input and output, tokens in and out, latency, finish reason
• Tool calls: name, arguments, result size, duration, error
• The run: turn count, total tokens, total cost, terminal state (completed / limit hit / error / refused)