Web Scraping

Tool Selection

• Firecrawl: general scraping, JS rendering (primary tool)
• Bright Data MCP: when site blocks normal scraping (anti-bot bypass)
• Direct WebFetch: simple pages without protection

Best Practices

• Respect robots.txt
• Rate limit requests (1-2 sec between)
• Cache results locally
• Structure output as JSON/CSV

Legal Notes

• Public data only
• Check ToS of target site
• Don't scrape personal data without consent