llm-eval-harness
Installation
SKILL.md
LLM Eval Harness
Overview
Give this skill an endpoint (base_url + model + an API key in an env var) and it
measures whether the endpoint actually works and whether the model is fast, stable,
protocol-correct, and good enough — instead of trusting the vendor's headline numbers.
Six dimensions, usually scattered across ad-hoc scripts that get rewritten (with the
same bugs) every time:
| Dimension | Script | Answers |
|---|---|---|
| Availability | scripts/availability_probe.py |
which model IDs work here, with 3-state error classification |
| Request fidelity | scripts/fidelity_probe.py |
do system prompt / tools / history actually REACH the model? |
| Speed | scripts/speed_probe.py |
TTFT + sustained decode tok/s, thinking-aware |
| Concurrency / stability | scripts/concurrency_probe.py |
success rate, p50/p90 latency, where it breaks |
| Protocol compliance | scripts/protocol_probe.py |
does the Anthropic thinking block actually fire when requested, AND does the endpoint accept one already sitting in history on a later turn (N≥10, both are separate checks — see Dimension 3)? |
| Quality / use-case regression | scripts/usecase_runner.py + blind judges |
does it pass your accumulated cases? |