Unified Framework for LLM Agent Evaluation

This skill enables Claude to design, build, and audit standardized evaluation harnesses for LLM-based agents following the framework proposed by Zhu et al. (2026). The core insight is that most agent benchmark results are confounded by extraneous factors -- system prompt differences, tool invocation formats, memory strategies, inference engine variability, and live environment drift -- rather than reflecting genuine model capability. This skill teaches how to isolate those factors by constructing a hermetically deterministic sandbox with a standardized dataset (Instruction Set, Tool Set, Environment Set), a unified agent architecture, and a multidimensional evaluation methodology covering outcome-level, process-level, and efficiency metrics.

When to Use

• When building an evaluation harness or benchmark suite for LLM-based agents and you need results that are reproducible and comparable across models.
• When auditing an existing agent benchmark for confounding factors (prompt leakage, tool-format bias, environment non-determinism).
• When comparing two or more agent frameworks (e.g., LangChain vs. smolagents vs. custom ReAct) and you need to isolate model performance from framework effects.
• When setting up CI/CD pipelines that gate deployments on agent benchmark scores and you need deterministic pass/fail criteria.
• When designing a tool-use benchmark and you need a standardized protocol for tool definitions, parameter types, and invocation formats.
• When debugging non-reproducible agent failures caused by live web environments, API version drift, or stochastic inference behavior.

Key Technique

The Problem: Confounded Benchmarks. Current agent evaluations mix together at least five independent variables: (1) inference configuration (provider safety filters, engine choice like vLLM vs. SGLang, floating-point non-associativity even at temperature=0), (2) prompting and planning strategy (lightweight vs. detailed system prompts, ReAct variants, reflection mechanisms), (3) memory formatting and management (structured vs. flat, FIFO truncation vs. summarization, long-term retrieval strategies), (4) tool invocation conventions (naming restrictions, unsupported parameter types across providers), and (5) external environment state (live URLs expiring, web content changing between runs). When a benchmark score changes, it is impossible to attribute the delta to the model vs. any of these factors.

The Solution: Hermetic Determinism via Sandbox + Standardized Dataset + Unified Architecture. The framework proposes three interlocking components. First, a Standardized Dataset comprising an Instruction Set (tasks with ground-truth criteria), a Tool Set (Python-based tool definitions using a single canonical schema), and an Environment Set (version-controlled, mocked replacements for all external dependencies). Second, a Unified Agent Architecture that fixes the non-model components -- inference wrapper, memory module, planner, tool executor -- so only the LLM itself varies between runs. Third, a Multidimensional Evaluation Methodology that goes beyond final-answer accuracy to measure outcome correctness (response + environment state diff against gold snapshot), process correctness (tool invocation trajectory vs. ground-truth path), robustness (pass@k with standardized k), efficiency (token count, latency, interaction steps), and failure taxonomy (categorized as reasoning, planning, tool-use, or environment errors).