ReSearch: Multi-Stage Intent-Aware Search Pipelines

This skill enables Claude to design and implement multi-stage search systems inspired by the ReSearch framework (arXiv:2601.14176). The core insight is that dataset and document discovery should be decomposed into three explicit stages -- intent interpretation, high-recall hybrid retrieval, and LLM-powered reranking -- rather than handled by a single retrieval pass. This separation lets each stage optimize for its own objective (coverage vs. precision) and handles the semantic gap between abstract user goals ("I want to study Arctic ice loss trends") and concrete metadata fields (variable names, platform codes, temporal ranges).

When to Use

• When the user asks to build a search system over a catalog of datasets, documents, or resources with heterogeneous metadata
• When the user needs to find scientific datasets matching a high-level research objective rather than a specific keyword
• When building a retrieval pipeline that must handle both precise keyword queries ("MODIS sea surface temperature") and abstract task queries ("analyze drought impact on crop yields")
• When the user wants to combine lexical search (BM25) with semantic embeddings and LLM reranking in a single pipeline
• When existing search returns too many irrelevant results or misses relevant items because metadata uses abbreviations, jargon, or indirect descriptions
• When the user wants to build an evaluation benchmark by extracting ground-truth dataset references from published papers

Key Technique

The Recall-Precision Separation Principle. Traditional search systems optimize a single relevance score, forcing a tradeoff between finding all relevant items (recall) and ranking them well (precision). ReSearch decomposes this into two explicit phases. The recall phase casts a wide net using multiple complementary retrieval strategies -- BM25 lexical matching, dense semantic embeddings, and abbreviation-expanded metadata. These are fused into a candidate pool of ~100 items. The precision phase then applies an LLM reranker that reads each candidate's metadata against the original query context and reorders the list. This two-phase design avoids premature filtering that would drop relevant but oddly-described items.

Intent Classification Drives Query Processing. Before retrieval begins, the system classifies each query as Type A (specific: mentions variables, platforms, or dataset names) or Type B (broad: expresses an abstract research goal). Type A queries get spell correction and direct retrieval. Type B queries are rewritten by an LLM into data-oriented search formulations that extract structured constraints (temporal range, spatial bounds, required variables) from the natural language intent. This rewriting step is critical -- experiments show that task-based queries degrade retrieval performance by 10-20% without it.