Discover failures with a human

Read the operating agreement, data sourcing, and the selected surface: managed or local review.

For bundled local tools and exact commands, read toolkit setup. Run the workspace installer for this skill; it preserves existing customized files.

Assemble evidence

Reuse existing traces and review notes. Seek production examples, complaints/incidents, support tickets, expert examples, and targeted synthetic variations in that order as practical. Ask about unresolved source access and retainability before pulling content. Inspect one real execution before a new managed connection.

Create a diverse discovery sample. Include ordinary traffic and random exploration alongside difficult examples; cluster only when it helps. Explain which dimensions the sample covers and which it misses. Fifteen to twenty-five examples can start a review, but stop/grow based on discovery saturation and domain breadth, not a universal count.

Review loop

1. Show full behavior in its natural form, not a raw JSON cell. Start with open-ended notes: what went wrong, what should have happened, and why it matters.
2. Save human observations immediately with stable annotation and trace IDs. Note provisional themes to guide further sampling; defer taxonomy consolidation and assignment review to the error-analysis stage.
3. Alternate breadth (new task types) and depth (more instances of a known failure). Revisit earlier examples when the meaning of failure changes.
4. After several human-reviewed records establish a category, optionally delegate one category per background analyzer to find candidates. Without delegation, scan sequentially. Reviewers accept or dismiss suggestions; suggestions never become human ground truth automatically.
5. Track confirmed versus suggested findings, reviewed coverage, and new-category discovery. Pause when further review mostly repeats known categories, then propose the next missing coverage or curation step.