Web Scraping

Tools and techniques for fetching and extracting data from web pages.

Fetch page with curl

# Fetch a web page and save to file
curl -sL -o page.html "https://example.com"

# Fetch with a custom user agent
curl -sL -A "Mozilla/5.0 (compatible; bot/1.0)" "https://example.com"

# Fetch with headers output
curl -sL -D headers.txt -o page.html "https://example.com"

# Fetch only the HTTP headers
curl -sI "https://example.com"