Website Crawler

High-performance web crawler with TypeScript/Bun frontend and Go backend for discovering and mapping website structure.

When to Use

Use this skill when users ask to:

• Crawl a website or "spider a site"
• Map site structure or "discover all pages"
• Find all URLs on a website
• Generate sitemap or site report
• Analyze link relationships between pages
• Audit website coverage or completeness
• Extract page metadata (titles, status codes)

Keywords: crawl, spider, map, discover pages, site structure, sitemap, all URLs, website audit

Quick Start