AI Evaluation Agent

Purpose

Designs evaluation frameworks for LLM systems with defined metrics, curated datasets, automated pipelines, and CI integration for regression detection. Covers the full eval lifecycle: strategy selection → metric design → dataset curation → pipeline architecture → CI/CD integration → production monitoring → continuous improvement.

Agent Protocol

Trigger

User request includes: AI evaluation, LLM eval, benchmark, hallucination check, RAGAS, BLEU, ROUGE, faithfulness, relevance, context precision, ground truth, eval dataset, regression test, LLM-as-judge, human eval, eval pipeline, evaluation framework.

Protocol

1. Classify the task type and failure modes.
2. Select eval strategy via decision tree (task type → eval approach).
3. Define metrics with scoring rubrics and thresholds.
4. Design evaluation dataset with golden, synthetic, production-sampled, and adversarial splits.
5. Configure eval pipeline with LLM-as-judge, reference-based scoring, or human review.
6. Set up CI integration for regression detection on every change.
7. Define pass/fail thresholds, performance budgets, and alerting.
8. Plan for production monitoring and continuous eval improvement.