Error Analysis

Two phases against sampled traces, spans, or sessions. Open coding reads each sampled entity and writes a short, specific note on what went wrong. Axial coding groups those notes into a few narrow annotations, each judging one dimension of the entity with a small label set, and counts the labels to feed eval design and fix prioritization. Open coding always comes first: dimensions and labels that are not grounded in notes are invented top-down, which is the failure this workflow exists to avoid. Axial coding can also start from any existing set of open-ended observations.

Recorded notes are the deliverable. Open coding ends with one note per problematic entity written to the server and mirrored in the local sidecar, not with a summary in chat. Reading traces and reporting what you noticed without recording anything is a read-only diagnosis, the top-down shortcut this workflow exists to replace. A project-wide question such as "find any issues", "what's going wrong", or "any errors here" is a request for phase 1: pick the identifier, pick the unit, record a note for each problem as you find it, then summarize. A project with no annotation names yet is the normal starting state, not a reason to hold off writing.

Choosing the unit of analysis

Pick trace, span, or session deliberately before recording — the choice determines what every note and annotation below targets, and a wrong default is expensive to undo mid-run. The unit is where the failure modes you're investigating actually live:

• Trace — one input → one output. Classifiers, single-shot summarizers, stateless tool-using agents, single-query RAG.
• Span — one operation inside a trace. In-isolation mechanical failures (exception, tool error response, malformed output) or failures attributable on sight to one component.
• Session — a sequence of traces sharing a session.id. Multi-turn agents where the failure is a trajectory — context loss across turns, goal drift, forgotten preferences — that exists only across traces.

Three signals to read:

1. User framing. "Conversation", "agent forgot", "drift", "across turns" → session. "This trace", "wrong output" → trace. "Exception", "malformed", "the retrieval failed" → span.

2. Session wiring. The session id lives at the root span's session.id attribute (not a top-level trace field); "" means absent. List ~200 recent traces and compute how many carry a non-empty session id, how many distinct session ids appear, and the median traces per session. No session ids or median 1 → trace or span level; median 2+ → session level is plausibly right.