NLWeb Retrieval Backends

Before writing code

Fetch live docs:

1. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/nlweb-retrieval.md for the architectural overview.
2. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/config/config_retrieval.yaml for the canonical list of endpoint names and their defaults — config keys move release to release.
3. Pick the per-backend setup page from docs/setup-*.md (Qdrant, Azure AI Search, Elasticsearch, OpenSearch, Postgres, Snowflake, Cloudflare AutoRAG).
4. Inspect AskAgent/python/retrieval_providers/<backend>.py for the exact client signature and required env vars.
5. Verify the embedding dimension and metric (cosine/dot/L2) the backend expects — must match the embedding provider.

Conceptual Architecture

The Read-Fanout, Single-Write Pattern

NLWeb does something unusual: it reads from every enabled retrieval endpoint in parallel and deduplicates by URL, but writes go to exactly one write_endpoint. This means:

• You can run a hybrid index (e.g., local Qdrant for site content + Bing for fresh news) without code changes
• You migrate between backends by re-running db_load against the new write_endpoint
• "Result quality" is the union of all enabled stores — a noisy backend pollutes the top-k