LLM Evaluation

Help users build, audit, and improve evaluation systems for LLM pipelines.

Where to Start

Have an existing eval pipeline? Start with an eval audit to surface problems: missing error analysis, unvalidated judges, vanity metrics. See the eval-audit reference.

Starting from scratch? Begin with error analysis on real traces. If no production data exists, generate synthetic data first. See the error-analysis and generate-synthetic-data references.

Test Suites

Test suites are the primary way to test agents in Opik. They combine test items with string assertions checked by an LLM judge, plus execution policies for multi-run reliability testing. Available in both Python and TypeScript SDKs.

Python:

import opik