AI crawler access

What this does and when

Answers one question per crawler: can it fetch this page, and is it allowed to use what it fetches for the job it does? Crawlers do different jobs, and mixing them up is the most common mistake in AI-search advice:

• Search crawlers decide whether a site can be shown and cited in an AI product's answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, and Googlebot/Bingbot, whose indexes feed Google's AI features and Copilot).
• Training crawlers and tokens decide whether content may train models (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Google-Extended, Applebot-Extended). Blocking these does not remove a site from AI search.
• User-triggered fetchers open a page because a person asked (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher). Several of them say robots.txt may not apply to them.