🍮 crawlbrulee map

map enumerates the urls on a site. it combines sitemap discovery with homepage link extraction, dedupes the result, and paginates it. use it to plan what to scrape.

new to crawlbrulee? read the crawlbrulee skill first. for the shared bits — auth, proxies, caching, errors — read crawlbrulee-api.

there is no crawl endpoint

map is half of the answer to "crawl this site". the other half is scraping each url you keep. the loop is yours to write:

1. map the site to get candidate urls, paginating until you have enough.
2. filter the list yourself — keep a path prefix, drop assets and anchors, cap the count.
3. scrape each survivor with the formats you want (see crawlbrulee-scrape).
4. assemble the results.

step 2 is the one that matters. map returns everything it finds; you decide what's worth scraping. scraping is where the cost is, so cap the list before you start — check usage first, and see https://crawlbrulee.com/pricing for the cost model.

the request