Webcrawl — build a RAG-ready Markdown corpus

Turn a docs site or domain into clean, chunk-friendly Markdown files for RAG and embeddings. webcrawl gives you URL discovery (map), scoped multi-page crawling (crawl), content-addressed caching, and SHA-256 content hashes for dedupe.

Quick start

# 1. Discover the URL space
npx webcrawl map https://docs.stripe.com --filter webhook -n 50

# 2. Crawl the scoped section into Markdown
npx webcrawl crawl https://docs.stripe.com/webhooks -n 25

# 3. If the site publishes llms.txt, use it as the canonical page list
npx webcrawl scrape https://modelcontextprotocol.io/llms.txt

Workflow