Automated Multi-Agent Scoring for Subjective Responses

This skill enables Claude to evaluate subjective, open-ended text responses (interview transcripts, essays, reflective writing, short answers) using a multi-agent decomposition strategy from Huynh et al. (2026). Instead of asking a single LLM pass to score everything at once, the technique splits evaluation into (1) a preprocessing/refinement stage that cleans and distills the response, then (2) parallel criterion-specific scoring agents, each calibrated with three strategically chosen examples (low/medium/high). This structured decomposition nearly doubles scoring agreement with human experts compared to fine-tuned baselines (QWK 0.62 vs 0.32) and generalizes across domains without retraining.

When to Use

• When the user wants to score or grade open-ended text responses against a rubric with multiple criteria (empathy, communication, ethical reasoning, argumentation, etc.)
• When building an automated evaluation pipeline for interviews, admissions essays, reflective journals, or short-answer assessments
• When a single-pass LLM scoring attempt produces inconsistent or criterion-conflated results
• When the user has a rubric with distinct scoring dimensions and wants per-criterion reliability
• When evaluating transcripts from spoken interviews (including noisy ASR output)
• When the user wants to replicate human panel scoring at scale without fine-tuning a model

Key Technique

Why single-pass scoring fails on subjective tasks. When an LLM scores a response on multiple criteria simultaneously, cross-criterion interference occurs: the model's judgment on one dimension (e.g., empathy) bleeds into its score for another (e.g., ethical reasoning). Fine-tuning on rationale-based methods also struggles because subjective qualities like empathy are expressed through implicit narrative signals rather than explicit keywords. The model learns surface patterns rather than the underlying construct.

The multi-agent decomposition. The framework uses a two-stage pipeline. First, a Refiner agent preprocesses the raw text: correcting transcription errors, stripping filler words and conversational preambles, removing redundancies, and distilling the core substantive content without adding anything new. Second, the refined text is distributed to N independent Scorer agents, one per rubric criterion. Each scorer receives only its own criterion definition, the scoring scale descriptors, and three calibrated exemplars selected at the 5th, 50th, and 95th percentiles for that specific criterion. This isolation prevents cross-contamination between criteria.