How to Check robots.txt: Testing for Blocks Now That the Search Console Tester Is Gone

The robots.txt Tester is gone, replaced by the “robots.txt report”

Search Console used to have a robots.txt Tester where you could edit the contents of robots.txt and test whether a URL would be blocked for Googlebot. Google announced its retirement in November 2023 and shut it down in December. What replaced it is the “robots.txt report” under “Settings”.

The report is for seeing which robots.txt Google is currently reading; you can’t test a draft before you change it. The URL Inspection tool tells you whether a live URL is blocked by robots.txt, but not which line is responsible. When you want to check the effect before editing, the quickest way is to paste your draft into a tester and run the check.

Where robots.txt lives and how to open it

robots.txt goes in the top level of the site with the name “robots.txt”. Open “https://your-domain/robots.txt” in a browser to see what’s currently published.

• There’s only one location: https://example.com/robots.txt is read, but a file placed lower down, such as https://example.com/blog/robots.txt, is not.
• Each host is separate: www.example.com, example.com and subdomains such as shop.example.com each need their own robots.txt.
• http, https and port numbers are separate too: the contents of http://example.com/robots.txt aren’t used for https:// pages.
• Use a lowercase file name: Robots.txt or ROBOTS.TXT may not be read.
• Use UTF-8: if you write non-ASCII paths or comments, save the file as UTF-8.

How to check blocking URL by URL with the tester

1. 1

   Copy and paste the contents of robots.txt

   Open https://your-domain/robots.txt, copy everything, and paste it into the tester’s “robots.txt contents”. If you’re planning changes, you can paste the unpublished draft instead.

2. 2

   Enter the URLs to check, one per line

   Both full URLs such as “https://example.com/blog/” and paths only such as “/blog/” work. Listing different kinds of URL, such as the home page, articles, categories, search results, images and PDFs, makes gaps easier to spot.

3. 3

   Choose the crawlers to check

   Start with Googlebot and Bingbot. Add Googlebot-Image if image search matters, AdsBot-Google and Mediapartners-Google if you run ads, and GPTBot or ClaudeBot if you’re deciding how to handle AI.

4. 4

   Read the results table and the deciding rule

   Each URL × crawler shows “Allowed” or “Blocked”. Open a URL to see the line number of the rule that decided it and the group (User-agent) that was used. If the result isn’t what you expected, review that line.

5. 5

   Fix the syntax check items and test again

   It flags, with line numbers, syntax that Google won’t read the way you intended, such as a full-width colon or a Disallow line containing a URL. After fixing, test the same URLs again and confirm the results before publishing.

The matching rules are explained in detail in Allow vs Disallow precedence, and what to watch for when adding a Googlebot group in how to write User-agent lines.

How long it takes Google to pick up changes

Google doesn’t start using a new robots.txt the moment you edit it. It normally caches the robots.txt it fetched for about 24 hours and keeps crawling with the old contents during that time.

• If you’re in a hurry: in Search Console’s robots.txt report, you can request a recrawl from the “︙” menu for that robots.txt.
• To confirm it’s applied: once the report’s “last crawled” time is after your edit and the contents shown are the new ones, it has been picked up.
• Search results change more slowly: blocking crawling doesn’t immediately remove URLs already in search results. Likewise, after removing a block it takes time for pages to be recrawled and regain their standing.

Also note that Google behaves differently when your server returns an error for robots.txt. “Not found” responses such as 404 are treated as “no restrictions (everything allowed)”, but if 5xx errors or timeouts continue, Google stops crawling the whole site for a while. You can check in the report’s fetch status that robots.txt alone isn’t erroring during maintenance.

Things that are easy to miss when checking

• Mixing up staging and production: make sure a “Disallow: /” written for a pre-launch site hasn’t been carried over to production. It’s a classic reason for a whole site gradually vanishing from search results.
• Blocking CSS and JavaScript: blocking /assets/ or /wp-includes/ can stop Google rendering pages properly, which may affect how mobile-friendliness and other factors are assessed.
• Upper and lower case: “Disallow: /Admin/” doesn’t block /admin/. Check that it matches how the URL is actually written.
• File size: Google only reads the first 500 KiB. If an auto-generated robots.txt runs to thousands of lines, rules near the end may be ignored.
• Combining with noindex: if you block a page you want removed from search results in robots.txt, the noindex is never read and the page stays. See robots.txt vs noindex for details.

FAQ

Where is the robots.txt Tester in Search Console?

It was retired in December 2023 and no longer exists. Instead, the report under “Settings” → “robots.txt” shows the contents Google fetched, when it fetched them and any errors. To check whether individual URLs are blocked, use the URL Inspection tool (live URLs only) or a robots.txt tester.

Do changes to robots.txt take effect immediately?

No. Google normally caches robots.txt for about 24 hours, so changes aren’t picked up right away. If you’re in a hurry, you can request a recrawl from the robots.txt report in Search Console.

What happens if there’s no robots.txt?

When robots.txt isn’t found (404), it’s treated as “no restrictions” and every URL can be crawled. If there are no pages you want to block, you don’t need a robots.txt at all.

Do subdomains need their own robots.txt?

Yes. robots.txt is read separately for each host, so rules for blog.example.com written in example.com/robots.txt aren’t used. Put one at the top level of each subdomain.