Site Crawler Skill

Respectfully crawl documentation sites and web content for RAG ingestion.

Overview

Documentation sites, blogs, and knowledge bases contain valuable structured content. This skill covers:

• Respectful crawling (robots.txt, rate limiting)
• Structure-preserving extraction
• Incremental updates (only fetch changed pages)
• Sitemap-based discovery

Prerequisites

# HTTP client
pip install httpx