AI bots allowed in robots.txt
What the check finds
robots.txt is missing or empty. Crawling still works, an absent file means everything is permitted, but there is nowhere to declare the sitemap or to control AI crawlers.
robots.txt blocks an answering bot, the crawlers that fetch a page in order to cite it. This is the direct way to be absent from AI answers.
robots.txt blocks a training crawler. Usually deliberate: most publishers now refuse training while admitting answering bots.
Why it matters for AI answers
Answering bots fetch pages to cite them in AI answers, so blocking them removes the site from those answers. Training crawlers are a separate choice that does not stop citation.
How to fix it
- Allow answering bots in robots.txt, one User-agent line each: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot.
- Decide separately on training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended.
- Note that Google-Extended does not affect Google Search or AI Overviews, which use Googlebot.
- Add a Sitemap: line with the full sitemap URL to robots.txt.
Related checks
Review this finding in context: AI access and guidance guide ›
Still stuck? Contact support ›