Web Scraper

Overview

Recursively scrape web pages with concurrent processing, extracting clean text content while following links. The scraper automatically handles URL deduplication, creates proper directory hierarchies based on URL structure, filters out unwanted content, and respects domain boundaries.

When to Use This Skill

Use this skill when users request:

• Scraping content from websites
• Downloading documentation from online sources
• Extracting text from web pages at scale
• Crawling websites to gather information
• Archiving web content locally
• Following and downloading linked pages
• Research data collection from web sources
• Building text datasets from websites

Prerequisites