Scraping Skill

Progressive Escalation

1. Direct fetch — curl/fetch with standard headers
2. Browser rendering — Headless browser for JS-heavy sites
3. Proxy rotation — For rate-limited or geo-restricted content
4. Specialized APIs — Platform-specific scrapers

Rules

• Respect robots.txt
• Rate limit requests
• Handle pagination
• Extract structured data
• Store raw + processed versions