Crawl Budget

Optimizes how search engines spend crawl capacity. Crawl budget only matters at scale — sites under roughly 10k indexable URLs rarely have a budget problem; say so if the site is small instead of manufacturing work.

Inputs

• Required: site URL and approximate page count
• Optional: server log samples, robots.txt, XML sitemap(s), list of known parameterized URL patterns

Data pulls

Fetch with the webfetch tool: