Link Crawler

Crawl root URLs to a configurable depth, verify each discovered link is active (HTTP 200 + real content), and present results as a grouped markdown list.

When to Use

• User asks to find all active pages on a site or section.
• User provides a list of URLs and wants to know which are alive.
• User wants a page inventory / sitemap / link tree scraped and reported.

When NOT to Use

• Fetching a single page's contents → just use a curl/WebFetch.
• Downloading a specific file (PDF, image, archive).
• Browsing requires login / authenticated session.
• The user wants to test if a site is up (one URL, no crawl) — just curl it.

Quick Reference