Exploring AI failures

The highest-value thing you can do with production AI traffic is look at where it fails and name the patterns. The catch: most failures are silent. The model returns a clean response — HTTP 200, no exception — that is wrong, off-topic, ignores an instruction, or misuses a tool. Those never raise an error, and they're usually the failures worth caring about.

So this skill is about finding failures (loud and silent), reading them, and grouping them into a ranked set of failure modes you can act on: fix a prompt, file a bug, prioritize work, or turn the top mode into an automatic eval (creating-online-evaluations).

Everything below serves one irreducible activity: reading real traces. The queries only tell you which traces to open — they are never the answer. If you report a list of problems without having opened traces, you've described the loud minority (the things that throw errors) and missed the job.

This is bottom-up: the failure modes emerge from real traces, not from a list of generic metrics decided in advance. For reading a single trace in depth, lean on exploring-llm-traces; for emergent grouping at high volume, exploring-llm-clusters.