pagination-and-crawling
Installation
SKILL.md
Pagination and Crawling
Two failure modes dominate: missing a third of the records without noticing, and crawling forever because the site generates infinite URLs.
Both are prevented by the same discipline — know how many records you expect, and bound the crawl explicitly.
Identify the pagination type first
| Type | Looks like | Reliability |
|---|---|---|
| Cursor / token | ?cursor=eyJpZCI6... |
Best. Stable under inserts |
| Keyset | ?after_id=1042&limit=50 |
Best. Same reason |
| Page number | ?page=3 |
Fine, but shifts if records are added mid-crawl |
| Offset | ?offset=100&limit=50 |
Same drift problem, plus slow on large tables |
| Link header / next URL | rel="next" in headers or HTML |
Follow it, don't construct URLs |
| Infinite scroll | No URL change | Backed by an API. Find it |
| "Load more" button | No URL change | Same |
Prefer cursor and keyset paging where offered. Page and offset paging drift: if three records are inserted while you're on page 4, records shift across the boundary and you silently miss some and duplicate others.