NLWeb Data Loading

Before writing code

Fetch live docs:

1. Fetch https://github.com/nlweb-ai/NLWeb/blob/main/docs/tools-database-load.md for the canonical db_load.py reference.
2. Inspect AskAgent/python/data_loading/db_load.py and db_load_utils.py in the live repo for exact CLI flags — they've added flags in recent releases.
3. Check AskAgent/python/data_loading/rss2schema.py for how RSS items map to Schema.org Article objects.
4. Confirm the embedding provider used at ingest matches preferred_provider in config_embedding.yaml for the query side — mismatch = silent retrieval failure.
5. For partner backends, check docs/setup-snowflake.md, docs/setup-cloudflare-autorag.md, etc. for backend-specific ingest steps (some bypass db_load.py).

Conceptual Architecture

What db_load Does

db_load.py is the canonical ingest pipeline. Given a source and a site name, it: