ScraperAPI DataPipeline

DataPipeline is a managed scraping product. You define a project (what to scrape, how often, where to send results), and ScraperAPI runs it on your schedule without you managing proxies, retries, or infrastructure.

When NOT to use DataPipeline

• One-off scrapes of a known URL list → use the Async API — faster, cheaper, no project setup.
• Exploring a site without known URLs → use the Crawler.
• Need results in real-time within your code → Async API is programmable; DataPipeline is scheduled.
• Free plan, need recurring execution → recurring schedules require a paid plan.

Use DataPipeline when: scraping runs on a fixed schedule, the input list is large (up to 100,000 items), results should flow to a webhook automatically, or you want email notifications on job completion.

Base URL and Auth