Creating online evaluations

An online evaluation automatically scores either each matching $ai_generation or the whole trace containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic, not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once those failure modes are identified and turns them into scoped, continuously-running evals.

One eval per failure mode, and as many evals as the data justifies. How many to create is a judgment call you make from what the traces actually showed — sometimes one, often three or four. Never assume the answer is one, and never bundle several modes into one evaluator.

Propose before you create. Bring the user a short list of candidate evals and let them pick which ones they want (Phase 1.1). Creating evals they didn't ask for costs them money and noise.

First, know what you're evaluating. Finding and ranking the failure modes worth catching is a separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague about it and don't refer to a specific failure mode, run exploring-ai-failures to scope a use case, find failing traces, and produce a ranked list of failure modes.