Web Scraping

Overview

Two ways to scrape in the sandbox:

1. MCP tools (scrape_page, scrape_pages) — recommended for straight "give me this page's content". Synchronous, return dicts.
2. Direct Scrapling Python API — for CSS/XPath selectors, sessions, logins, and multi-page spiders. Async, returns Page objects with .css() / .xpath().

Quick fetches can run inline via ExecuteCode. For spiders, multi-URL crawls, or anything you'll iterate on, write the scraper to <task_name>/scraper.py and run it via Bash — edit-and-rerun beats resubmitting code.

MCP Tools

Import from tools.scrape. Synchronous — no await.

from tools.scrape import scrape_page, scrape_pages