MLLM-Eval Skill

Purpose

This skill makes an LLM / MLLM clinical evaluation defensible: a real adjudicated reference standard, faithfulness measured not assumed, clinical-efficacy metrics beyond n-gram overlap, a pretraining- contamination check, prompt-sensitivity disclosed, and a reader study where text is generated. It is model-agnostic — every check applies to a closed API and to open weights — and read-only (an advisory design/audit skill): it audits the evaluation design and specifies and routes the clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 via their published extractors) rather than running the model or computing the metrics itself.

It is the LLM/MLLM evaluation-design counterpart in the lane — an auditor that hands the specified metrics to their extractors and /analyze-stats, parallel to how /model-validation audits an imaging model's design (the imaging metrics themselves are computed by /model-evaluation). The reviewer-side audit of a finished manuscript uses the mllm_evaluation.md (ME0–ME8) probe via /self-review and /peer-review; this skill is the author-side harness design. It routes the reader study to /design-ai-benchmarking, the sizing to /calc-sample-size, and TRIPOD-LLM / MI-CLEAR-LLM compliance to /check-reporting.