Crawlee

Use this skill when work touches Crawlee crawlers: choosing a crawler, routing and link discovery, storage, anti-blocking, scaling, runtime/Docker/Apify deployment, or v3 to v4 migration.

Snapshot (2026-10-09): crawlee@3.18.2 is latest (stable). crawlee@4.0.0-rc.1 is the rc tag; the v4 tag moves daily (4.0.0-beta.225). Check source-map.md and the installed version before answering. Examples below use v3 names; v4 differences are in v4-migration.md.

Workflow

1. Inspect first: crawlee major/minor, sub-packages (@crawlee/impit-client, @crawlee/jsdom), playwright/puppeteer versions, Node/Bun, ESM vs CJS, existing router/routes, storage dir, proxy provider, Docker base image, Apify usage.
2. Pick the cheapest crawler that returns the data (table below; details in crawler-choice.md). Prove the data is absent from raw HTML before reaching for a browser. Also check for a JSON/XHR endpoint or sitemap.
3. Route work to the focused references:
   • Crawler choice, HTTP clients, impit impersonation, parsers: crawler-choice.md.
   • Router, labels, enqueueLinks, queues, lists, sitemaps, robots.txt: routing-requests.md.
   • Dataset, KeyValueStore, purge behavior, config/env vars, storage backends: storage-config.md.
   • Sessions, proxies, fingerprints, blocking, retries, autoscaling, politeness, anti-blocking checklist: anti-blocking-scaling.md.
   • Node vs Bun, Docker, Apify Actor SDK: runtimes-deploy.md.
   • Version status and v3 to v4 upgrade: v4-migration.md.
4. Keep the project's existing crawler, router file layout, and deployment topology unless asked to migrate.
5. Run a small capped crawl (maxRequestsPerCrawl: 10) before any large run. Read the statistics log, not only the output.