RAG Auditor

Systematic RAG pipeline evaluation across the full retrieval-generation chain: designs evaluation query sets, measures retrieval metrics (Precision@K, Recall@K, MRR), evaluates generation quality (groundedness, completeness, hallucination rate), diagnoses component-level failures, and recommends targeted improvements.

Reference Files

File	Contents	Load When
`references/retrieval-metrics.md`	Precision@K, Recall@K, MRR, NDCG definitions and calculation	Always
`references/generation-metrics.md`	Groundedness, completeness, hallucination detection methods	Generation evaluation needed
`references/failure-taxonomy.md`	RAG failure categories: retrieval, generation, chunking, embedding	Failure diagnosis needed
`references/diagnostic-queries.md`	Designing evaluation query sets, known-answer questions, difficulty levels	Evaluation setup

Prerequisites

Access to the RAG pipeline (or its outputs for post-hoc evaluation)

rag-auditor

RAG Auditor

Reference Files

Prerequisites

More from mathews-tom/armory

architecture-reviewer

architecture-diagram

youtube-analysis

concept-to-image

concept-to-video

code-refiner