AI crawler access check

One wrong line in robots.txt removes a site from AI answers completely, and no amount of content work can compensate. This check takes under a minute and should run before any other AI-visibility work.

Scope

Read https://<domain>/robots.txt and nothing else. Do not crawl the site, do not attempt to access disallowed paths, and do not bypass any access control. This is a read of one public file.

Procedure

1. Fetch

Fetch https://<domain>/robots.txt.