HTML Extraction Patterns

Most scrapers break for the same reason: they were written against the page that existed on the day, using selectors tied to how it happened to look. Six weeks later a designer changes a class name and the extraction silently returns nothing, or worse, returns the wrong thing.

The fix is choosing selectors by how stable they are, not by what the browser inspector offers when you right-click.

Look for structured data first

Before writing a single selector, check whether the page already publishes what you want. A surprising share of the web does.

# JSON-LD is often the entire record, cleanly typed
curl -s https://example.com/product/123 \
  | grep -oPz '(?s)<script type="application/ld\+json">.*?</script>' \
  | sed -e 's/<[^>]*>//g'

# Open Graph and meta tags
curl -s https://example.com | grep -oE '<meta[^>]*(og:|twitter:|name="description")[^>]*>'