AI Evaluation

Purpose

Design an evaluation framework for AI/LLM features, including golden dataset creation, automated scoring rubrics, hallucination detection, and regression testing infrastructure.

Scope Constraints

Reads feature specifications, evaluation requirements, and existing infrastructure details for framework design. Does not execute model inference, create production datasets, or access live model endpoints directly.

Inputs

• AI feature being evaluated (what it does, expected behavior)
• Input data examples and edge cases
• Quality requirements (accuracy thresholds, hallucination tolerance)
• Existing evaluation infrastructure (if any)
• Production monitoring requirements

Input Sanitization