ISD-Agent-Bench: Building Theory-Grounded Instructional Design Agents

This skill enables Claude to design, build, and evaluate LLM-based agents that automate Instructional Systems Design (ISD) using the ADDIE framework (Analysis, Design, Development, Implementation, Evaluation). The core insight from ISD-Agent-Bench is that integrating classical ISD theories with ReAct-style reasoning — specifically using coarse-grained phase-level tools rather than fine-grained sub-step tools — achieves the highest performance (86.49 vs. 82.96 for fine-grained). This skill teaches how to construct Context Matrix scenarios for systematic coverage, implement theory-grounded ISD agents, and apply multi-judge evaluation protocols to eliminate LLM-as-judge bias.

When to Use

• When the user asks to build an agent or pipeline that generates course curricula, training programs, or lesson plans
• When designing educational content that must systematically cover analysis, design, development, implementation, and evaluation phases
• When the user wants to generate diverse instructional scenarios by crossing learner characteristics, domains, delivery modes, and constraints
• When evaluating the quality of LLM-generated educational materials using structured rubrics
• When building a ReAct-style agent that needs to follow a multi-phase structured methodology (not limited to education — the pattern generalizes)
• When the user needs to set up multi-judge LLM evaluation with inter-rater reliability checks

Key Technique

Context Matrix Scenario Generation. The paper's Context Matrix framework crosses 51 contextual variables across 5 categories (Learner Characteristics, Institutional Context, Educational Domain, Delivery Mode, Constraints) with 33 ISD sub-steps derived from ADDIE. This produces combinatorially diverse scenarios that stress-test an agent's ability to adapt instructional design to specific contexts. Scenarios are generated via a five-stage pipeline: seed collection from domain literature, stratified sampling of variable combinations, LLM-based content generation, two-layer quality control (rule-based field validation + LLM consistency review), and targeted augmentation of underrepresented categories.

React-ADDIE: Coarse-Grained Theory + ReAct Reasoning. The top-performing architecture (React-ADDIE) wraps each ADDIE phase as a single tool call rather than decomposing into 14 fine-grained sub-step tools. At each phase, the agent generates a reasoning thought, executes the phase tool, and observes the result — standard ReAct loop. The key finding: coarse-grained decomposition (5 tools) outperforms fine-grained (14 tools) because it lets the LLM handle intra-phase reasoning natively rather than forcing rigid sub-step sequences. This principle — match tool granularity to the LLM's natural reasoning span — applies broadly to agent design.