How to Diagnose Bad Search Quality

Before diagnosing or evaluating search quality, establish two different ground truths: a labeled query set measures whether the results are the right ones; exact KNN measures whether ANN finds what brute force would (no labels needed). The first diagnoses relevance, the second diagnoses the index.

Need a Labeled Baseline to Score Quality or Validate a Gain

Use when: user has no golden set, asks "how do I know if my search is good?", or needs to gate releases on a retrieval metric. Every fix in the sections below should be validated this way.

• Pick the metric by usage: Recall@k for RAG, MRR/Hits@1 for single-answer, NDCG@k for re-ranking Choosing the metric
• Build a labeled query set (human, log-based, or LLM-synthetic) and score retrieval with ranx Measuring Retrieval Relevance
• When tuning and evaluating search quality, set a minimum target gain, then make sure your labeled query set is large enough to measure it reliably. Quantify the uncertainty around a measured gain with a confidence interval (e.g. bootstrap the per-query gains, 95%): if it includes zero, the improvement is inconclusive and may be due to query-to-query variation. Adding labeled queries narrows the interval; how many you need depends on the size of the gain you are targeting and on query-to-query variation, not on collection size Before tuning a collection
• If you use the labeled queries to tune or select a configuration, evaluate the final choice on held-out queries. A gain that doesn't survive the held-out evaluation isn't reliable evidence of an improvement.
• For full RAG pipelines, also score generation with Ragas and use the retrieval-vs-generation 2x2 to isolate regressions Pipeline Output Quality
• Gate CI on a per-metric threshold to catch regressions from embedding-model swaps, prompt changes, or index config changes

Don't Know What's Wrong Yet

Use when: results are irrelevant or missing expected matches and you need to isolate the cause.