Error Analysis

Guide the user through reading LLM pipeline traces and building a catalog of how the system fails.

Overview

1. Collect ~100 representative traces
2. Read each trace, judge pass/fail, and note what went wrong
3. Group similar failures into categories
4. Label every trace against those categories
5. Compute failure rates to prioritize what to fix

Core Process

Step 1: Collect Traces

Capture the full trace: input, all intermediate LLM calls, tool uses, retrieved documents, reasoning steps, and final output.

Target: ~100 traces. This is roughly where new traces stop revealing new kinds of failures. The number depends on system complexity.