How to allow AI crawlers like GPTBot in robots.txt
Your robots.txt must not block the main AI crawlers, such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended, or those assistants cannot cite your site. Audit each user-agent group, remove Disallow rules for bots you want, and keep blocks only for scrapers you reject.
By ratemysite.ai editorial team. Updated .
How do you fix it?
- Open https://yourdomain/robots.txt and read each User-agent group, including the wildcard * group.
- Look for Disallow: / rules aimed at GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot.
- Remove the rules for the assistants you want to be cited by, and keep blocks for scrapers you reject.
- Check that your CDN or firewall does not block these user agents with a bot-fight setting.
- Re-run the audit and confirm the AI bots check passes.
What are AI crawlers?
An AI crawler is an automated program, run by an AI company, that fetches web pages to train models or to answer user questions with live sources. Examples are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot and Google-Extended. Each identifies itself with a user-agent name in robots.txt. If you block the bot that powers an assistant, that assistant has no way to read or cite your pages. A single line such as Disallow: / under a GPTBot group blocks that bot from every page, and a security plugin can add 1 or more of these groups without anyone noticing.
How do accidental blocks happen?
The usual causes are a template robots.txt copied from a blog post that blocks all AI bots, a CMS plugin with a block AI toggle, or a CDN rule that challenges unknown user agents. Fix them by reading the file line by line and testing a fetch with the bot's user-agent string. Decide on purpose: allow assistants that send you traffic and citations, and block those you do not want. Document the decision so the next redesign keeps it.
How much does this check matter?
- This check is worth about 3.6% of the overall score.
- A finished report is reused for 24 hours before you can re-analyze it.
- Each page is fetched with a timeout of 10 seconds, and HTML larger than 2 MB is not audited.
What else do people ask?
- Is allowing AI bots the same as allowing training?
- Not always. Some vendors use separate bots for search and for training. Read each vendor page and allow the search bot even if you block the training one.
- Does blocking Google-Extended affect Google Search?
- No. Google-Extended controls use of content for Gemini AI training and grounding. It does not change how Googlebot crawls for Search.