web-scraper
Installation
SKILL.md
CLI
python3 /home/workspace/Skills/web-scraper/scripts/scraper.py <command> [options]
| Command | Description |
|---|---|
scrape <url> |
Fetch page content as markdown |
scrape <url> --css <selector> |
Extract elements matching a CSS selector |
scrape <url> --schema <json-file> |
Extract structured data using a JSON CSS schema |
scrape <url> --llm-extract "<instruction>" |
Use LLM to extract structured data (Crawl4AI) |
crawl <url> --depth N --max-pages N |
Deep crawl following links (BFS) |
batch <file> [--css <selector>] |
Scrape multiple URLs from a file (one per line) |