crw-map — URL discovery without content

When to use

• You need to know which URLs exist on a site before committing to a crawl — map first, then crawl only the section you need.
• Step 3 in the crw ladder: if you want content, combine with crw-scrape (single pages) or crw-crawl (bulk). Map is URL-only — no page content is returned.
• Estimating crawl cost: crw map docs.example.com | wc -l tells you how many pages a subsequent crawl would touch before you commit.
• Finding a specific path: filter the URL list with grep instead of crawling the whole site.

Quick start