Crawlee

Overview

Crawlee is a web scraping and crawling library that handles the hard parts — request queuing, retries, proxy rotation, browser fingerprinting, and rate limiting. Use Cheerio for fast HTML-only scraping or Playwright/Puppeteer for JavaScript-rendered pages. Built-in storage for datasets, request queues, and key-value stores. Scales from single pages to millions of URLs.

When to Use

• Scraping data from websites (product prices, job listings, articles)
• Crawling entire sites for content or link analysis
• JavaScript-rendered pages (SPAs, React/Vue sites)
• Scraping at scale with proxy rotation and anti-blocking
• Structured data extraction with automatic retries

Instructions

Setup