Technical SEO Audit

Categories

robots.txt: exists, valid, not blocking important resources
XML sitemap: run claude-seo run sitemap_discovery.py <url> --json; require a valid entry in found, and report stale or unsafe robots.txt declarations separately from working fallback locations
Noindex tags: intentional vs accidental
Crawl depth: important pages within 3 clicks of homepage
JavaScript rendering: check if critical content requires JS execution
Crawl budget: for large sites (>10k pages), efficiency matters
Googlebot fetch limits: Googlebot fetches the first 2MB of HTML and first 64MB of a PDF (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
Crawl rate auto-adjusts (backs off on 5xx/slow responses); there is no manual crawl-rate control (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
Google's canonical crawling/robots reference moved to developers.google.com/crawling (migrated 2025-11-20); IP-range files relocated to /crawling/ipranges/ and googlebot.json was renamed common-crawlers.json.