Creating online evaluations

An online evaluation automatically scores either each matching $ai_generation or the whole trace containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic, not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once that failure mode is identified and turns it into a scoped, continuously-running eval.

First, know what you're evaluating. Finding and ranking the failure modes worth catching is a separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague about it and don't refer to a specific failure mode, run exploring-ai-failures to scope a use case, find failing traces, and produce a ranked list of failure modes.

For the mechanics of writing and iterating an evaluator (Hog source vs LLM-judge prompt, dry-running, debugging a live eval), defer to exploring-llm-evaluations.

Tools