trace-annotation-tool
Trace Annotation Tool Generator
Generate a custom local web application for open coding of LLM traces — the first qualitative pass of error analysis in the Analyze phase of the evaluation lifecycle.
Core Workflow
Step 1: Understand the User's Trace Data
- Ask the user to point to their trace data file (CSV, JSONL, JSON, or any structured format).
- Read a sample of the data to understand its structure: field names, nesting depth, which fields represent the user query, intermediate steps, tool calls, and final output.
- Identify a unique trace identifier field (or generate sequential IDs if none exists).
- Confirm the structure with the user: "I see fields X, Y, Z — which represent the trace steps, and which is the user query?"
Step 2: Ask About Additional Features
The tool includes these features by default:
More from maragudk/evals-skills
llm-as-a-judge
>
12failure-taxonomy
>
4prompt-engineering
Use this skill when crafting, reviewing, or improving prompts for LLM pipelines — including task prompts, system prompts, and LLM-as-Judge prompts. Triggers include: requests to write or refine a prompt, diagnose why an LLM produces inconsistent or incorrect outputs, bridge the gap between intent and model behavior, reduce ambiguity in instructions, add few-shot examples, structure complex prompts, or improve output formatting. Also use when the user needs help distinguishing specification failures (unclear instructions) from generalization failures (model limitations), or when iterating on prompts based on observed failure modes. Do NOT use for general coding tasks, document creation, or non-LLM writing.
4