Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems

This skill enables Claude to build retrieval-augmented multi-agent pipelines that automatically generate instance-specific evaluation rubrics for medical dialogue systems. Rather than relying on generic metrics or expensive expert annotation, the technique retrieves authoritative medical evidence, decomposes it into atomic facts (positive assertions, contraindications, safety red flags), extracts interaction intent constraints from the user query, and synthesizes both tracks into weighted, verifiable evaluation criteria. The resulting rubrics can score LLM responses, discriminate between subtly different answer qualities, and guide targeted response refinement.

When to Use

• When the user needs to evaluate medical chatbot or clinical QA system outputs against evidence-based criteria rather than surface-level metrics like BLEU or ROUGE.
• When building an automated evaluation pipeline for HealthBench-style medical benchmarks where physician-authored rubrics are unavailable or too costly.
• When the user wants to detect subtle clinical errors (wrong dosage, missing contraindication, omitted red flag) that generic evaluators miss.
• When implementing a rubric-guided response refinement loop where an LLM's medical answer is iteratively improved against structured criteria.
• When the user asks to score and rank multiple medical LLM responses with discriminative sensitivity (near-miss detection between answers differing by one clinical fact).
• When designing a multi-agent framework where specialized agents handle retrieval routing, fact decomposition, intent extraction, rubric synthesis, and auditing.

Key Technique

The core insight is dual-track constraint construction: medical evidence and user interaction intent are processed in parallel by specialized agents, then merged into a single rubric. The objective track retrieves content from authoritative sources (CDC, WHO, PubMed, Mayo Clinic, drug databases), synthesizes overlapping snippets, and decomposes them into three categories of atomic facts: positive facts (declarative assertions, dosage ranges, conditional logic), negative constraints (explicit prohibitions, contraindications), and safety red flags (emergency warnings). The subjective track extracts explicit instructions and implicit communication cues from the user query, identifying medically necessary but missing contextual variables (e.g., patient age, comorbidities not stated).

These two tracks feed into a Rubric Synthesis Agent that maps each atomic fact and interaction constraint to a structured criterion tuple (criterion_text, evaluation_axis, clinical_weight). Each criterion is assigned to one of five axes: accuracy (factual correctness, safety violations), completeness (topic coverage), context_awareness (clarifying questions for missing info), communication_quality (tone, empathy, clarity), and instruction_following (formatting constraints). Clinical weights range from -10 to +10 following severity tiers: safety-critical items get extreme weights (|8-10|), completeness items get moderate weights (|4-7|), and minor details get low weights (|1-3|). Negative weights penalize harmful content.