Overview

The workflow consists of 3 main phases in a loop:

1. Data Ingestion - Transform raw agent log data into an ingested format for analysis
2. Feedback Configuration and data exploration - Refine what and how we want to provide feedback on agent transcripts by exploring the raw data via traditional file search tools and automated llm-as-a-judge based filtering
3. Annotation - Annotate the transcript data based on the feedback configuration

NOTE: use the multichoice/checklist/survey tool as much as possible when getting feedback from the user, typing is onerous.

Setup and Housekeeping

This document covers setup tasks that should be completed before starting the annotation workflow.

Prerequisites

This skill requires Python and Node.js dependencies. If the user encounters errors about missing packages, ask for permission to install dependencies: