crw-crawl — bulk page extraction

When to use

• You need content from many pages under a site or section, not just one.
• Step 4 in the crw ladder: if you only need a handful of known URLs, use crw-scrape in a loop instead — it's simpler and gives you content immediately. Use crawl when the set of URLs is unknown or large.
• Always map first (crw-map) to estimate page count before committing. A misconfigured crawl on a 50 000-page site is expensive; a map call is cheap.
• Start conservative: depth 1, limit 10. Scale up once you verify scope.

Quick start