Website Crawler API - Map, Crawl and Batch Extract to Markdown

Crawl a whole website and get every page back as clean Markdown, plain text or HTML; list a site's URLs from its sitemap; or extract a list of URLs in one job. Three operations on the Extract family:

• Map - a site's page URLs from its sitemaps and start page. Synchronous, 1 credit.
• Crawl - discover and extract one site's pages breadth-first as a job. Billed per successful page.
• Batch - extract up to 1,000 URLs you already have as a job. Billed per successful page.

For a single page, use the url-to-markdown-api skill (POST /api/v1/extract) instead.

When to trigger

Use this skill when the user asks to:

• Crawl a website, a docs site, a blog or a help center
• Turn a whole site (or a section of it) into Markdown for an LLM, a vector store or RAG
• List every URL on a site, read its sitemap, or find pages under a path
• Convert a sitemap into Markdown
• Scrape or extract a list of URLs in one go (batch scrape)
• Build a knowledge base or keep one fresh from a site