Blocking AI Crawlers with robots.txt: GPTBot, ClaudeBot and More

Think of AI crawlers in three groups

Most AI companies use a different crawler name for each purpose. Because robots.txt lets you decide name by name whether to block, it helps to understand the roles first.

• Training: crawlers that collect public pages to build AI models. Blocking them makes it less likely your pages end up in future training data.
• AI search: crawlers that collect pages so an AI can introduce or cite sites in its answers, such as ChatGPT search or Perplexity. Blocking them makes it less likely you’ll be mentioned in those answers.
• User-requested fetches: an agent that opens a page on the spot when someone asks the AI to “read this page”. Some companies treat these as outside the scope of robots.txt.

The main AI crawlers and their roles

These names are published by each company, and new crawlers appear and names change from time to time. Check each company’s official documentation for the current names before you block anything.

How to write robots.txt for each goal

1. 1

   Block AI training only, stay in search and AI search

   Write six lines, User-agent: GPTBot / User-agent: ClaudeBot / User-agent: Google-Extended / User-agent: Applebot-Extended / User-agent: meta-externalagent / User-agent: CCBot, followed by a single “Disallow: /”. Leave the User-agent: * group as it is. This is the approach with the least side effects.

2. 2

   Also block citations in AI search

   Add OAI-SearchBot, Claude-SearchBot and PerplexityBot to the names above. Your site will be mentioned less often in ChatGPT or Perplexity answers, so think carefully if you expect traffic from AI.

3. 3

   Keep only certain folders out of training

   Instead of “Disallow: /”, list only the folders you want to protect, such as “Disallow: /members/” and “Disallow: /illust/”. That way you can, for example, allow blog posts but keep illustrations and paid articles out.

In the tester’s “Create AI crawler blocks”, you can generate these lines in one go by choosing “Block training only”, “Block training and AI search” or “Block all”. Add the lines to your robots.txt, then use the check to confirm the result is what you intended, such as Googlebot allowed and GPTBot blocked, before you publish. How to group several names together is explained in how to write User-agent lines.

Effects on Google Search and AI Overviews

• Blocking Google-Extended does not affect your rankings or appearance in Google Search.
• AI Overviews in Google Search are treated as part of Google Search, so Google-Extended can’t switch them off. If you don’t want to appear there, use page-level controls such as nosnippet.
• Blocking Googlebot will gradually remove you from Google Search itself, so don’t block Googlebot as an anti-AI measure.
• Blocking Applebot-Extended does not affect Applebot, which is used for Siri and Spotlight search.

What robots.txt can and can’t do

robots.txt is a request saying “please don’t fetch these pages”, and it can’t be enforced. It works on major crawlers that publish their names and state that they follow robots.txt, but it can’t stop crawlers that fake their name or ignore robots.txt.

• Data that has already been used for training is not withdrawn when you add lines to robots.txt. It only affects future fetching.
• Your robots.txt can’t stop copies of your text republished on other sites, or anything already included in datasets such as Common Crawl in the past.
• If you need a hard block, you can also refuse requests from specific User-agents or IP addresses in your server or CDN settings.
• Removing a page from search results is not the same as stopping crawling. The difference is explained in robots.txt vs noindex.

FAQ

How do I stop ChatGPT from training on my site?

Add “User-agent: GPTBot” and “Disallow: /” to robots.txt. OpenAI’s training crawler will stop fetching your pages. OAI-SearchBot, which is used for ChatGPT search, has a different name, so leave it alone if you still want to appear in search.

Will blocking Google-Extended lower my Google rankings?

No. Google-Extended only decides whether your content may be used for Gemini and other AI. It is not used for Google Search crawling or ranking.

If I block AI crawlers, will I disappear from AI search?

If you block only the training crawlers such as GPTBot and ClaudeBot, the AI search crawlers OAI-SearchBot and Claude-SearchBot can still fetch your pages. If you also block the AI search names, you’ll be mentioned less often in those answers.

Does robots.txt guarantee my site won’t be used for AI training?

No. robots.txt is a request that crawlers choose whether to honour. The major companies say their crawlers follow it, but it can’t stop data that has already been collected or copies on other sites.