Online Eval Engineering

Build online evaluators iteratively:

inspect traces and interview user -> propose directions -> user chooses
-> build evaluator -> test, attach, verify -> review and repeat

Read references/langsmith-api.md before creating or modifying evaluators.

1. Inspect traces

Ask the user for their LangSmith project name. Fetch recent root-level traces and print their structure. Read references/trace-inspection.md. Find:

• run name and type;
• available input and output field names;
• the shape and content of the data (truncated samples);
• which fields carry the data an evaluator would need.