Eval Chat Traces

Systematically review AI chat traces to find failure patterns using error analysis methodology (inspired by Hamel Husain's evals framework).

Input

Optional: time range (default: last 7 days), sample size (default: 30 traces).

Process

Phase 1: Pull Traces

1. Query CloudWatch Logs Insights on /aws/lambda/extralife-ai-engagement-agent for ChatQuery logs.
2. Query for ChatFeedback logs from /ecs/ExtraLifeWebAdmin.
3. Prioritize: negative feedback traces first, then outliers (response_time_ms > P90, empty tools_invoked, error responses), then random sample to fill remaining.
4. For Athena deep dives: SELECT * FROM extralife_chat_logs.chat_logs WHERE ...

Phase 2: Present Traces for Review

Present traces in a structured table for efficient binary judgment: