ScraperAPI Crawler

The Crawler discovers and scrapes linked pages automatically, starting from a seed URL and following links that match your regex pattern. Use it when you need data from many pages of a site but don't have the URLs upfront.

When NOT to use the Crawler

• You have a known URL list → use the Async API instead; it's cheaper and more predictable.
• You need a single page → use the Standard API (api.scraperapi.com).
• The content is behind login → the Crawler only accesses publicly available pages.
• Free plan, need depth > 1 → free accounts are capped at max_depth: 1.

Endpoints