Where did the Search Console robots.txt Tester go?

The robots.txt Tester in Google Search Console was discontinued in December 2023 and replaced by the “robots.txt report” under Settings. The report shows whether robots.txt could be fetched and which lines have errors, but it has no feature for entering a URL to test whether it is blocked. For URL-by-URL judgments you can use this tool, or the URL Inspection tool (for published URLs only).

If both Allow and Disallow match, which one takes priority?

Google uses the rule with the longest (most specific) pattern. For example, with “Disallow: /blog/” and “Allow: /blog/public/”, /blog/public/a.html is allowed because the longer Allow wins. On a tie in length, Allow (the less restrictive one) is used. The order in which they are written does not matter.

What happens if I write both User-agent: * and User-agent: Googlebot?

Googlebot uses only its own group and never uses the rules of the “User-agent: *” group. If you want Googlebot to also be blocked from paths blocked for “*”, you must write the same Disallow lines in Googlebot's group too.

What do * and $ mean in robots.txt?

“*” matches zero or more of any characters, and “$” means “end of URL” only when placed at the end of a pattern. “Disallow: /*.pdf$” blocks URLs ending in .pdf and does not block URLs that continue with a query, such as /a.pdf?download=1. A trailing “*” on a pattern has no effect, so “/fish*” is the same as “/fish”.

Is there a difference between Disallow: /fish and Disallow: /fish/?

Yes. “/fish” is a prefix match, so it also blocks /fish.html and /fishheads. “/fish/” blocks only what is inside the folder and does not block /fish or /fish.html.

Will blocking in robots.txt remove a page from search results?

Not necessarily. robots.txt only stops crawling, so URLs linked from other pages may still appear in search results without any content. To remove a page from search results, add noindex to the page and do not block that page in robots.txt. If it is blocked, Googlebot cannot read the noindex.

How can I stop AI such as ChatGPT or Claude from training on my site?

In robots.txt, block OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended, Apple's Applebot-Extended, Common Crawl's CCBot and others with “Disallow: /”. Blocking Google-Extended and Applebot-Extended does not affect how your site appears in Google Search or Apple search. OAI-SearchBot, Claude-SearchBot and PerplexityBot are crawlers used for AI search answers, so blocking them makes your site less likely to be mentioned in those answers.

Can I slow down Googlebot's crawling by writing Crawl-delay?

No. Google does not read Crawl-delay. Googlebot adjusts its frequency automatically based on your server's response speed and errors. Bing and Yandex do read Crawl-delay.

How should I write non-ASCII URLs, such as Japanese ones, in robots.txt?

Google treats them the same whether you write them as-is, like “Disallow: /カテゴリ/”, or percent-encoded, like “Disallow: /%E3%82%AB%E3%83%86%E3%82%B4%E3%83%AA/”. Save the file as UTF-8. This tool also judges both ways of writing as the same URL.

robots.txt Tester | Check Whether URLs Are Blocked, Validate Syntax, Generate AI Crawler Blocks

A robots.txt checker you can use now that Search Console's robots.txt Tester is gone. Paste your robots.txt and the URLs you want to check, and it shows for each crawler — Googlebot, Googlebot-Image, AdsBot-Google, Bingbot, Applebot, and AI crawlers such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot — whether it is allowed or blocked, plus the deciding rule (with line number). Like Google, it follows these rules: the longest matching rule wins and Allow wins on a tie; if a group names the crawler, User-agent: * is not used; * and a trailing $; and non-ASCII URLs. It also checks syntax (full-width colons, URLs in Disallow, rules before User-agent, crawler-name typos such as Googlebot-Images, the unsupported Noindex, Crawl-delay that Google ignores), generates a robots.txt that blocks only AI training, and includes a cheat sheet for * and $.

1. Paste your robots.txt

Open “https://your-domain/robots.txt” on your site, copy its contents, and paste them here. You can also try a draft before publishing it. Results use the same reading as Google (the longest matching rule wins; if equal in length, Allow wins).

Samples:

robots.txt contents

5 lines, 111 bytes (Google reads only the first 500 KiB), 1 groups, 1 Sitemaps

2. URLs and crawlers to check

One URL per line (a full URL starting with https:// or just a path like /blog/ both work). Up to 50 are checked.

URLs / paths (one per line)

Google

Other search engines

3. Results

URL	Googlebot	Googlebot-Image	Google-Extended	Bingbot	GPTBot	ClaudeBot
https://example.com/	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed
https://example.com/wp-admin/	Blocked	Blocked	Blocked	Blocked	Blocked	Blocked
https://example.com/wp-admin/admin-ajax.php	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed
https://example.com/wp-admin-guide/	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed

https://example.com/Path used for the check: /

Allowed
GooglebotGoogle Search
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Allowed
Googlebot-ImageGoogle Images
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
If there is no Googlebot-Image group, it follows the Googlebot group.
Allowed
Google-ExtendedUse in Gemini training and answers
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Google-Extended is not a crawler; it is a name that decides whether pages fetched by Googlebot may be used for generative AI such as Gemini. Blocking it does not affect Google Search rankings or display.
Allowed
BingbotBing Search
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Allowed
GPTBotOpenAI training
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Allowed
ClaudeBotAnthropic (Claude) training
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)

https://example.com/wp-admin/Path used for the check: /wp-admin/

Blocked
GooglebotGoogle Search
Matched “Disallow: /wp-admin/” (line 2).
Group used: User-agent: * (line 1)
Blocked
Googlebot-ImageGoogle Images
Matched “Disallow: /wp-admin/” (line 2).
Group used: User-agent: * (line 1)
If there is no Googlebot-Image group, it follows the Googlebot group.
Blocked
Google-ExtendedUse in Gemini training and answers
Matched “Disallow: /wp-admin/” (line 2).
Group used: User-agent: * (line 1)
Google-Extended is not a crawler; it is a name that decides whether pages fetched by Googlebot may be used for generative AI such as Gemini. Blocking it does not affect Google Search rankings or display.
Blocked
BingbotBing Search
Matched “Disallow: /wp-admin/” (line 2).
Group used: User-agent: * (line 1)
Blocked
GPTBotOpenAI training
Matched “Disallow: /wp-admin/” (line 2).
Group used: User-agent: * (line 1)
Blocked
ClaudeBotAnthropic (Claude) training
Matched “Disallow: /wp-admin/” (line 2).
Group used: User-agent: * (line 1)

https://example.com/wp-admin/admin-ajax.phpPath used for the check: /wp-admin/admin-ajax.php

Allowed
GooglebotGoogle Search
Of the matching rules, the longest one, “Allow: /wp-admin/admin-ajax.php” (line 3), is used. The order they are written in does not matter.
Group used: User-agent: * (line 1)
Other matching rules: Disallow: /wp-admin/ (line 2)
Allowed
Googlebot-ImageGoogle Images
Of the matching rules, the longest one, “Allow: /wp-admin/admin-ajax.php” (line 3), is used. The order they are written in does not matter.
Group used: User-agent: * (line 1)
Other matching rules: Disallow: /wp-admin/ (line 2)
If there is no Googlebot-Image group, it follows the Googlebot group.
Allowed
Google-ExtendedUse in Gemini training and answers
Of the matching rules, the longest one, “Allow: /wp-admin/admin-ajax.php” (line 3), is used. The order they are written in does not matter.
Group used: User-agent: * (line 1)
Other matching rules: Disallow: /wp-admin/ (line 2)
Google-Extended is not a crawler; it is a name that decides whether pages fetched by Googlebot may be used for generative AI such as Gemini. Blocking it does not affect Google Search rankings or display.
Allowed
BingbotBing Search
Of the matching rules, the longest one, “Allow: /wp-admin/admin-ajax.php” (line 3), is used. The order they are written in does not matter.
Group used: User-agent: * (line 1)
Other matching rules: Disallow: /wp-admin/ (line 2)
Allowed
GPTBotOpenAI training
Of the matching rules, the longest one, “Allow: /wp-admin/admin-ajax.php” (line 3), is used. The order they are written in does not matter.
Group used: User-agent: * (line 1)
Other matching rules: Disallow: /wp-admin/ (line 2)
Allowed
ClaudeBotAnthropic (Claude) training
Of the matching rules, the longest one, “Allow: /wp-admin/admin-ajax.php” (line 3), is used. The order they are written in does not matter.
Group used: User-agent: * (line 1)
Other matching rules: Disallow: /wp-admin/ (line 2)

https://example.com/wp-admin-guide/Path used for the check: /wp-admin-guide/

Allowed
GooglebotGoogle Search
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Allowed
Googlebot-ImageGoogle Images
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
If there is no Googlebot-Image group, it follows the Googlebot group.
Allowed
Google-ExtendedUse in Gemini training and answers
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Google-Extended is not a crawler; it is a name that decides whether pages fetched by Googlebot may be used for generative AI such as Gemini. Blocking it does not affect Google Search rankings or display.
Allowed
BingbotBing Search
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Allowed
GPTBotOpenAI training
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)
Allowed
ClaudeBotAnthropic (Claude) training
No rule in the User-agent: * group matches, so it is allowed.
Group used: User-agent: * (line 1)

4. Syntax check

Reports lines Google cannot read, meaningless lines, and common misunderstandings, with line numbers.

Error: 0Warning: 0Info: 0

No syntax problems were found.

List of groups

*
From line 1, 1 Disallow, 1 Allow

Sitemap

Line 5: https://example.com/wp-sitemap.xml

The pasted robots.txt and URLs are only saved in this browser and are not sent to the server. The tool also does not access your site to fetch robots.txt.

ABOUT

About robots.txt Tester | Check Whether URLs Are Blocked, Validate Syntax, Generate AI Crawler Blocks

The robots.txt Tester lets you paste your site's robots.txt and judge, per crawler, whether each URL you want to check is “allowed” or “blocked”. The robots.txt Tester in Google Search Console was discontinued in December 2023, and the current “robots.txt report” cannot judge individual URLs, so this tool was made to fill that gap. Judgments follow Google's specification (RFC 9309 and the behavior of the robots.txt parser Google has published): when both Allow and Disallow match, the rule with the longest pattern is used, and Allow is used on a tie. “*” is treated as zero or more of anything and a trailing “$” as the end of the URL, paths are case-sensitive, and non-ASCII URLs are normalized to the %E3%81… form before comparing. It also reflects that Googlebot does not use the “User-agent: *” rules if a group names Googlebot, that Googlebot-Image follows the Googlebot group if it has no group of its own, that AdsBot-Google and AdSense's Mediapartners-Google ignore “*”, and that Applebot follows the Googlebot rules. You can also pick AI crawlers such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and CCBot, and there is a feature to generate a robots.txt that blocks only AI training. The syntax check reports, with line numbers, full-width colons, Disallow lines containing a URL, rules before User-agent, crawler-name mistakes such as Googlebot-Images, the unsupported Noindex, Crawl-delay that Google does not read, and a Sitemap given as a path only.

How to use robots.txt Tester | Check Whether URLs Are Blocked, Validate Syntax, Generate AI Crawler Blocks

1Open “https://your-domain/robots.txt” on your site, copy its contents, and paste them into “robots.txt contents”
2Enter the URLs or paths you want to check, one per line (a full URL starting with https:// or just a path like /blog/ both work)
3Select the crawlers you want to judge, such as Googlebot, Bingbot, GPTBot, and ClaudeBot
4In the results table, check “Allowed” or “Blocked” for each URL × crawler
5Open a URL to see the line number of the deciding rule and the group (User-agent) that was used
6Fix the error and warning lines in the “Syntax check”, then run the judgment again
7To block AI crawlers, create the lines with “Create AI crawler blocks”, add them to the robots.txt under test, and verify

Great for

The Search Console robots.txt Tester is gone, and you want an alternative to check whether a URL is blocked
Before changing your robots.txt, you want to confirm you are not accidentally blocking important pages
You want to actually try your own rules to see whether Allow or Disallow takes priority
You want to verify that wildcard rules such as “Disallow: /*?” or “Disallow: /*.pdf$” behave as intended
You want to exclude only search result pages or URLs with parameters from crawling
You want to create a robots.txt that blocks only AI training crawlers such as GPTBot, ClaudeBot and Google-Extended
After adding a Googlebot group, you want to confirm the “User-agent: *” rules have not stopped applying to it
You want to confirm that wp-admin's admin-ajax.php is allowed in a WordPress robots.txt
When taking over a blog or site, you want to audit the robots.txt the previous owner wrote for mistakes

Good to know

Judgments follow Google's way of reading. Other search engines such as Bing, and AI crawlers, may differ in the details.
When both Allow and Disallow match, the longer (more specific) pattern is used, and Allow is used on a tie. The approach of reading top to bottom and using the first matching rule is not used.
If there is even one group that names Googlebot, Googlebot uses only that group's rules and does not use the “User-agent: *” rules. Paths you want blocked in the same way as “*” must also be written in Googlebot's group.
Googlebot-Image, Googlebot-Video and Googlebot-News follow the Googlebot group if there is no group with their own name, and then “*”. AdsBot-Google and Mediapartners-Google (AdSense) ignore “*”, so to block them you must write their names.
Paths are case-sensitive. “Disallow: /Admin/” does not block /admin/. Non-ASCII paths are normalized to the %E3%81… form before comparing, so either way of writing is judged as the same URL.
robots.txt stops crawling (fetching pages), not appearing in search results. If a blocked URL is linked from other pages, it may still appear in search results without a description. To remove a page from search results, use noindex and do not block that page in robots.txt (if blocked, noindex cannot be read).
Google ended support for Noindex in robots.txt on September 1, 2019. Google also does not read Crawl-delay.
robots.txt must be placed at the top of each domain (subdomains, http vs. https, and port numbers are all separate). Placing it somewhere like /blog/robots.txt means it will not be read.
Google reads only the first 500 KiB of robots.txt. Rules after that are ignored.
The names (User-agent) of AI crawlers are published by each company. New names may be added or changed, so check each company's guidance before blocking. robots.txt is a request that crawlers may or may not honor, and it has no enforcement power.
The pasted robots.txt and URLs are judged only inside this browser and are not sent to the server. The tool also does not access your site to fetch robots.txt.

Questions this tool answers

Use robots.txt Tester | Check Whether URLs Are Blocked, Validate Syntax, Generate AI Crawler Blocks for searches like these:

I want a tool to use instead of the Search Console robots.txt Tester
I want to check whether a specific URL is blocked by robots.txt
I want to know the priority between Allow and Disallow in robots.txt
I want to check how to use * and $ in robots.txt
I want to know what happens when both User-agent: * and Googlebot are written
I want to check my robots.txt for mistakes
I want to know how to write a robots.txt that blocks GPTBot and ClaudeBot
I want to stop only AI training and still appear in search
I want to know whether noindex or crawl-delay in robots.txt has any effect

FAQ

Where did the Search Console robots.txt Tester go?: The robots.txt Tester in Google Search Console was discontinued in December 2023 and replaced by the “robots.txt report” under Settings. The report shows whether robots.txt could be fetched and which lines have errors, but it has no feature for entering a URL to test whether it is blocked. For URL-by-URL judgments you can use this tool, or the URL Inspection tool (for published URLs only).
If both Allow and Disallow match, which one takes priority?: Google uses the rule with the longest (most specific) pattern. For example, with “Disallow: /blog/” and “Allow: /blog/public/”, /blog/public/a.html is allowed because the longer Allow wins. On a tie in length, Allow (the less restrictive one) is used. The order in which they are written does not matter.
What happens if I write both User-agent: * and User-agent: Googlebot?: Googlebot uses only its own group and never uses the rules of the “User-agent: *” group. If you want Googlebot to also be blocked from paths blocked for “*”, you must write the same Disallow lines in Googlebot's group too.
What do * and $ mean in robots.txt?: “*” matches zero or more of any characters, and “$” means “end of URL” only when placed at the end of a pattern. “Disallow: /*.pdf$” blocks URLs ending in .pdf and does not block URLs that continue with a query, such as /a.pdf?download=1. A trailing “*” on a pattern has no effect, so “/fish*” is the same as “/fish”.
Is there a difference between Disallow: /fish and Disallow: /fish/?: Yes. “/fish” is a prefix match, so it also blocks /fish.html and /fishheads. “/fish/” blocks only what is inside the folder and does not block /fish or /fish.html.
Will blocking in robots.txt remove a page from search results?: Not necessarily. robots.txt only stops crawling, so URLs linked from other pages may still appear in search results without any content. To remove a page from search results, add noindex to the page and do not block that page in robots.txt. If it is blocked, Googlebot cannot read the noindex.
How can I stop AI such as ChatGPT or Claude from training on my site?: In robots.txt, block OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended, Apple's Applebot-Extended, Common Crawl's CCBot and others with “Disallow: /”. Blocking Google-Extended and Applebot-Extended does not affect how your site appears in Google Search or Apple search. OAI-SearchBot, Claude-SearchBot and PerplexityBot are crawlers used for AI search answers, so blocking them makes your site less likely to be mentioned in those answers.
Can I slow down Googlebot's crawling by writing Crawl-delay?: No. Google does not read Crawl-delay. Googlebot adjusts its frequency automatically based on your server's response speed and errors. Bing and Yandex do read Crawl-delay.
How should I write non-ASCII URLs, such as Japanese ones, in robots.txt?: Google treats them the same whether you write them as-is, like “Disallow: /カテゴリ/”, or percent-encoded, like “Disallow: /%E3%82%AB%E3%83%86%E3%82%B4%E3%83%AA/”. Save the file as UTF-8. This tool also judges both ways of writing as the same URL.

Special Thanks

SOLO GUILD | サービス作るAIエンジニア

Back to all tools