Agent Studio Evaluations

Evaluation and eval-driven improvement workflows for Algolia Agent Studio agents, powered by the algobot evals command group of the algobot CLI (npm install -g algobot-ai).

This skill owns the workflows: what to measure, how to build datasets, how to tune an agent safely. The CLI mechanics (install, auth, command syntax, global flags, agent/chat/profile management) live in the algobot-cli skill: load it for any command-level question, or see references/evaluations.md for the full command map.

Version gate

Requires algobot-ai >= 2.1.0 (npm latest), which ships the evals command group and the agents/conversations fixes these workflows rely on. Probe with algobot evals metrics list; if the command is unknown, the installed algobot-ai predates evals: stop and install the release that ships it (npm i -g algobot-ai@latest). Verify with algobot --version.

Workflow 1: Evaluate an agent / create evaluation datasets

An evaluation measures an agent against a dataset of cases using judged metrics (profiles -> dataset -> experiment). Two dataset sources:

• Generated (default): experiments create resolves-or-creates a dataset grounded on the index and agent profiles. Zero authoring. Best for a first pass.
• Imported: evaluate on the agent's actual user traffic. One row per real user turn, prior turns frozen as context. Best when the baseline/candidate delta must be credible on real distribution.

For the real conversation history recipe (export -> transform -> PII scrub -> profiles -> atomic import -> pilot), the required expected keys per metric, and every pitfall (100-conversation pagination cap with the long-form --page flag, --jq quoted output stripping, ANSI/NO_COLOR piping on conversations list --json, filtering evalcase- / eval-experiment_run_* artifacts out of the history, parts[] to content flattening), read references/evaluations.md, section "Real conversation history datasets (import)".