agent-self-evaluation-patterns
Installation
SKILL.md
Agent Self-Evaluation Patterns
Overview
AI agents that cannot evaluate their own output quality are unreliable in production. A model that confidently produces wrong answers, fabricates citations, or never flags uncertainty becomes a liability rather than an asset. Self-evaluation patterns give agents structured mechanisms to detect errors, express calibrated uncertainty, and improve output quality before results reach users.
When to use: Designing agents that produce factual claims, code, analysis, or structured data; reviewing agent pipelines for hallucination risk; building eval suites for AI-generated content; any system where incorrect LLM output has meaningful downstream consequences.
Quick Reference
| Pattern | Core Problem | Key Technique | Failure Mode |
|---|---|---|---|
| Confidence Scoring | Agent returns wrong answers with false certainty | Logprob analysis, self-consistency sampling | Overconfident scoring — high score on hallucinated output |
| Chain-of-Thought Reflection | Errors baked into first draft go unchallenged | Generate → critique → revise cycle | Rubber-stamp reflection — critique that validates the original uncritically |
| LLM-as-Judge | Model cannot objectively evaluate its own output | Separate judge call with scoring rubric | Same model judging itself — no independence, shared biases |
| Hallucination Self-Detection | Claims unverifiable against source material | Source grounding checks, API existence verification | Reflection loop amplifies fabricated details instead of catching them |
| Output Quality Verification | Schema or structural errors in generated output | Assertion-based checking, schema validation | Checking format only — passes schema but semantically wrong |
| Eval-Driven Development | No objective measure of agent improvement | Define graders before implementation, regression gates | Graders written after the fact, shaped to pass existing output |