Model Evaluator

The Model Evaluator skill helps you rigorously assess and compare machine learning model performance across multiple dimensions. It guides you through selecting appropriate metrics, designing evaluation protocols, avoiding common statistical pitfalls, and making data-driven decisions about model selection.

Proper model evaluation goes beyond accuracy scores. This skill covers evaluation across the full spectrum: predictive performance, computational efficiency, robustness, fairness, calibration, and production readiness. It helps you answer not just "which model is best?" but "which model is best for my specific use case and constraints?"

Whether you are comparing LLMs, classifiers, or custom models, this skill ensures your evaluation methodology is sound and your conclusions are reliable.

Core Workflows

Workflow 1: Design Evaluation Protocol

1. Define evaluation objectives:
   • Primary goal (accuracy, speed, cost, etc.)
   • Secondary constraints
   • Failure modes to test
   • Real-world conditions to simulate
2. Select appropriate metrics:

   Task Type	Primary Metrics	Secondary Metrics