evaluations
Installation
SKILL.md
Set Up Evaluations for Your Agent
LangWatch Evaluations is a comprehensive quality assurance system. Understand which part the user needs:
| User says... | They need... | Go to... |
|---|---|---|
| "test my agent", "benchmark", "compare models" | Experiments | Step A |
| "monitor production", "track quality", "block harmful content", "safety" | Online Evaluation (includes guardrails) | Step B |
| "create an evaluator", "scoring function" | Evaluators | Step C |
| "create a dataset", "test data" | Datasets | Step D |
| "evaluate" (ambiguous) | Ask: "batch test or production monitoring?" | - |
Where Evaluations Fit
Evaluations sit at the component level of the testing pyramid — they test specific aspects of your agent with many input/output examples. This is different from scenarios (end-to-end multi-turn conversation testing).