Creating online evaluations

An online evaluation automatically scores either each matching $ai_generation or the whole trace containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic, not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once those failure modes are identified and turns them into scoped, continuously-running evals.

One eval per failure mode, and as many evals as the data justifies. How many to create is a judgment call you make from what the traces actually showed — sometimes one, often three or four. Never assume the answer is one, and never bundle unrelated modes into one evaluator. The exception is a set of modes that are outcomes of the same check, such as the reasons a reply missed the user's request: one categorical eval with a category per mode fits that (2.1).

Propose before you create. Bring the user a short list of candidate evals and let them pick which ones they want (Phase 1.1). Creating evals they didn't ask for costs them money and noise.