Python Web Scraping

Knowledge base for building production-grade Python web scraping systems. Covers the full stack from target assessment through production observability.

First Tool Call on Every Scraping Task

This section overrides everything else in this skill if there is any conflict. Read it first, act on it first.

When this skill activates on a scraping task, your next non-question tool call MUST launch a visible browser with the capture surface attached. Not Write pyproject.toml. Not Write models.py. Not "let me sketch the architecture first". Browser first, then code.

The one carve-out is a target listed under Bundled Targets below. Its Discovery Gate has already been run and its capture is recorded, so reading that reference and running its script satisfies this rule without reopening a browser. Anything not on that list gets the browser.

The default path is user-driven navigation with live capture, not Claude-clicks. The user knows their data and their portal better than you do, and authenticated SaaS sites need them anyway. Steps:

1. Ask the bare minimum to start: target URL, what data is wanted, authenticated yes/no. One short batch of questions, then stop asking.
2. Immediately open the browser: with the Playwright MCP tools of the playwright plugin (a declared dependency of this plugin, on Claude Code installed with claude plugin install playwright@claude-plugins-official, on other hosts by registering npx @playwright/mcp@latest as the MCP server playwright), browser_navigate to the target, which opens a visible browser, then dump with browser_network_requests once the user is done; or, when the target needs stealth or the Capture Surface below needs WebSocket, SSE or worker traffic, write an inline Patchright script via Bash that runs with headless=False, attaches every handler in the Capture Surface, and parks on input() waiting for the user.
3. Tell the user verbatim: "Browser is open with full network capture (XHR + fetch + WebSocket + SSE + workers + cookies + main-frame navigations). Log in, navigate to the data, apply the filters you'd use day-to-day, then press Enter here so I can dump the capture and reason from real endpoints."
4. While the user navigates, you watch the capture stream. When they press Enter you have: real URLs, real endpoint paths, real field names, real WebSocket frames, real auth cookies. Now you can scaffold.