seo-crawlability (M1)

Crawl access is the precondition for every other search signal: if Googlebot/Bingbot can't fetch the page and its assets, nothing else ranks. This module covers general-purpose crawl access only. AI-specific bot directives (GPTBot, Claude-SearchBot, etc.) live in seo-ai-crawlers (M14) and AI discovery files (llms.txt, agents.md, UCP) in seo-ai-discovery (M21); see references/ai-crawlers.md for that boundary.

Inputs

Work from the PageSnapshot named in your dispatch envelope: read parsed from <run_dir>/pages/<slug>.json plus status_chain and headers, and the parsed robots file in <run_dir>/site/robots.json (syntax, groups, Googlebot/Bingbot verdicts for the audited URL, Sitemap: lines); Grep pages/<slug>.html for verbatim evidence. Deterministic findings already emitted by audit.mjs are listed in <run_dir>/findings.deterministic.json — do not re-emit those ids; add model-judged findings only. If invoked directly with a URL/path and no snapshot exists, first run node "${CLAUDE_PLUGIN_ROOT}/scripts/snapshot.mjs" <target> --out "${CLAUDE_PLUGIN_DATA}/runs" and use the printed snapshot path.

Audits

Working from the PageSnapshot (parsed_rendered when render.used is not none, else parsed) plus site/robots.json:

Reachability: /robots.txt returns 200 (a 404 means "allow all" but is worth flagging; a 5xx can suspend crawling).
Syntax: each line is a valid directive (User-agent, Disallow, Allow, Sitemap, Crawl-delay); flag unknown tokens, missing User-agent group headers, and BOM/encoding issues.
Asset blocking: any Disallow that blocks CSS/JS, fonts, or /wp-includes/-style paths — this breaks rendering and is a leading cause of "page looks broken to Google" (cross-check with M-render).
Content blocking: Disallow rules that hide important indexable paths from Googlebot/Bingbot.
Crawl-delay sanity: a large Crawl-delay (or one applied to the global group) can starve crawl budget; note that Googlebot ignores Crawl-delay but Bingbot honors it.
Sitemap directive: presence of at least one absolute Sitemap: URL.
Overall access: resolve the effective ruleset for Googlebot and Bingbot against the audited URL — does it end up allowed?