crawl-websites-at-scale
Scrapy Web Scraping Skill
Scrapy is a fast, high-level Python web crawling and scraping framework. It enables structured data extraction from websites, supports crawling entire sites, and integrates pipelines to process and store scraped data.
When to use
- Crawl entire websites or follow links across many pages
- Extract structured data (prices, articles, product listings) into JSON/CSV
- Run scheduled or large-scale scraping pipelines
- Need built-in support for request throttling, retries, and middlewares
Required tools / APIs
- No external API required
- Python 3.8+ required
- Scrapy: Web crawling and scraping framework
Install options:
More from besoeasy/open-skills
news-aggregation
Aggregate and deduplicate recent news from multiple sources into concise topic summaries.
149free-geocoding-and-maps
Geocode addresses and get map data using free OpenStreetMap Nominatim API. Use when: (1) Converting addresses to coordinates, (2) Reverse geocoding coordinates to addresses, (3) Location-based features, or (4) Validating addresses.
63get-crypto-price
Fetch current and historical crypto prices and compute ATH or ATL over common time windows.
49anonymous-file-upload
Upload and host files anonymously using decentralized storage with Originless and IPFS.
47trading-indicators-from-price-data
Compute common trading indicators from OHLCV price data for analysis and strategy development.
46json-and-csv-data-transformation
Transform data between JSON, CSV, and other formats with filtering, mapping, and flattening. Use when: (1) Converting API responses to CSV, (2) Processing data pipelines, (3) Extracting specific fields, or (4) Flattening nested structures.
44