Turn annotations into failure modes

Read the operating agreement and analysis procedure and artifacts. For local work, use local review guidance; for an existing managed workflow, use managed guidance.

For bundled local tools and exact commands, read toolkit setup. Run the workspace installer for this skill; it preserves existing customized files. For the managed route, use concrete service operations after confirming the connected schema.

Start from reviewed evidence

Locate existing annotations, their source traces, sampling rules, and any taxonomy revisions. Preserve original notes and author/status information. Inspect the behavior behind the notes before grouping it. If there are no human observations, prepare a review queue and use eval-discover if installed; agent-generated observations remain suggestions.

Reuse a managed platform's existing analysis and exports when available. For either local track, perform the analysis over local records and extend the existing review UI. No grader framework or new hosted connection is required.

Propose and refine

Group notes by the observable failure and expected behavior. Use semantic search or embeddings only to find candidate neighbors; inspect evidence to decide membership. Similar wording can describe different failures, and different wording can describe the same failure.

Give each proposed mode a stable ID, clear definition, inclusion/exclusion rules, representative evidence, and boundary examples. Allow multiple modes per annotation and an unresolved queue. Separate observed symptoms from hypothesized root causes. Do not force every record into a category or turn a broad label such as “hallucination” into a supposedly actionable taxonomy.