Mapping URLs

crawlberg map <url> discovers the URLs a site exposes without rendering or extracting any page content. It reads sitemap.xml (including nested sitemaps), then falls back to link extraction from the seed page. Use it to plan a crawl, audit a site's surface, or feed a URL list into another tool.

Quick recipe

crawlberg map https://example.com --limit 500 --search docs --format markdown