web-scraping

Reliable scraping patterns for static HTML and JavaScript-rendered pages.

Overview

Start with requests + BeautifulSoup for static pages. When content is rendered by JavaScript, switch to Playwright. Always use timeouts, retries, and explicit user-agent headers.

When to Use

• Extracting structured data from HTML pages
• Monitoring content changes
• Handling SPAs or dynamic content with a headless browser

When Not to Use

• The site forbids scraping in ToS/robots.txt
• An official API is available (prefer it)