Crawler access audit

Goal: decide, for each crawler request in a log, who really made it, which rules applied at that time, and whether it was allowed.

1. robots.txt semantics (RFC 9309)

• File is a list of groups. A group starts with one or more User-agent lines followed by rules (Allow, Disallow, Crawl-delay, Sitemap). Blank lines and comments do not end a group; a new User-agent line after rules starts a new group.
• A crawler uses exactly ONE group: the one whose User-agent token best matches its product token (case-insensitive, longest match wins). If none matches, use the * group. Groups are NOT merged across different tokens, and * rules do not apply to a crawler that has its own group. If multiple groups name the same agent, merge those groups.
• Match the product token only (e.g. the name before /version), not the whole user-agent string.
• Path matching is on the URL path plus query string, starting at /, case-sensitive, prefix-based. * matches any sequence of characters, $ anchors the end of the URL. Rules are matched against the percent-decoded form of unreserved characters; compare after normalising percent-encoding consistently on both sides (encoded slashes stay encoded).
• Precedence: among all matching Allow and Disallow rules in the chosen group, the MOST SPECIFIC rule wins, measured by longest pattern length (octets). On a tie, Allow wins. An empty Disallow: allows everything. No matching rule means allowed.
• /robots.txt itself is always fetchable. Missing file (404) means everything allowed; server errors (5xx) mean assume disallow; unreachable for long means treat as allowed after caching limits.
• Crawl-delay is not part of RFC 9309; some crawlers honour it, some ignore it. Report violations only if the task asks, comparing the gap between consecutive requests from the same crawler to the delay.
• robots.txt is time-versioned: judge each request against the version in effect at the request timestamp (a change applies from its effective time, not retroactively). Crawlers also cache robots.txt for up to 24 hours, so be explicit if the spec asks for grace periods.

2. Page-level directives

• <meta name="robots" content="noindex, nofollow"> and X-Robots-Tag response headers control indexing and snippets, not crawling. A URL blocked in robots.txt cannot have its noindex seen. Header directives can target a specific bot (X-Robots-Tag: botname: noindex). They apply to non-HTML files too. When directives conflict the most restrictive applies.
• Do not conflate "disallowed to crawl" with "noindex".