STOP. READ THIS ENTIRE SKILL.MD BEFORE CALLING ANY ENDPOINT.

LLM Eval Lab

Evaluate and compare LLM models for task-specific accuracy, cost, and latency.

Split from /prompt-lab to keep prompt optimization separate from model evaluation. prompt-lab focuses on: system prompt in → optimized prompt out. llm-eval-lab focuses on: which model is cheapest/fastest/most accurate for a given task.

Quick Start

cd .pi/skills/llm-eval-lab

# Question-by-model comparison grid (best for finding minimum viable model)
./run.sh grid-eval -g example_qwen_comparison.json