robots.txt vs noindex: The Right Way to Remove a Page from Search Results
“I blocked it in robots.txt, but it still shows up in search.” “Search Console says ‘Indexed, though blocked by robots.txt’.” Most of these problems come from mixing up what robots.txt and noindex do. This article explains the difference between crawling and indexing, the correct steps to remove a page from search results, why Noindex and Crawl-delay in robots.txt don’t work, and how to deal with common Search Console messages and syntax mistakes.
robots.txt stops crawling, noindex stops indexing
Before a search engine shows a page in its results, there are two stages: crawling, where it fetches the page, and indexing, where it stores the content in its search database. robots.txt stops the first stage, crawling. noindex stops the second, indexing.
| robots.txt Disallow | noindex (meta tag / X-Robots-Tag) | |
|---|---|---|
| What it stops | Crawling (fetching the page) | Indexing (appearing in search results) |
| Where it goes | The site’s /robots.txt | The HTML head of each page, or an HTTP header |
| Appears in results? | If linked from elsewhere, the bare URL may still appear | Disappears once the page has been crawled |
| Best used for | Cutting wasted crawling, such as internal search results and parameter URLs | Keeping member pages, thank-you pages and duplicates out of search results |
| PDFs and images | Can be blocked by path | Use the X-Robots-Tag header |
Even if a page is blocked by robots.txt, Google still knows the URL exists when other pages link to it. So it may appear in search results with no description, without Google ever reading its content. That’s why a page can show up in search even though you blocked it.
The correct steps to remove a page from search results
- 1
Remove the robots.txt block
Google can only read noindex after it crawls the page. If the page stays blocked in robots.txt, noindex is never read and the page stays in the results. First use the tester to confirm the page is “Allowed” for Googlebot.
- 2
Add noindex to the page
Put <meta name="robots" content="noindex"> in the HTML head. For files that aren’t HTML, such as PDFs and images, configure the server to send an X-Robots-Tag: noindex header.
- 3
Wait for, or request, a recrawl
Requesting indexing with the URL Inspection tool in Search Console helps the page get read sooner. If there are many pages, keeping them in your sitemap for a while makes them more likely to be crawled.
- 4
Once they’re gone, block in robots.txt only if needed
After the pages have dropped out of the results, you can block them in robots.txt if you also want to reduce crawl load. But once blocked, noindex can no longer be read, so if other sites link to them the bare URL may reappear. In most cases it’s best to rely on noindex alone.
Common Search Console messages and what to do
| Message | Meaning | What to do |
|---|---|---|
| Indexed, though blocked by robots.txt | Crawling is blocked, but the URL was indexed from links and the like | If you don’t want it in search, unblock it and add noindex. If you do, just unblock it |
| Blocked by robots.txt | Crawling is blocked and the page is not indexed | No action needed if intended. If you want it in search, remove the matching Disallow |
| Submitted URL blocked by robots.txt | A URL listed in your sitemap is blocked by robots.txt | Remove it from the sitemap or unblock it |
| Excluded by ‘noindex’ tag | noindex was read and the page is excluded from results | No action needed if intended |
Search Console doesn’t tell you which Disallow line is responsible. Put the URL into the tester and check it as Googlebot to see the line number of the rule that decided the result. For how changes take effect after you edit the file, see how to check robots.txt.
Noindex and Crawl-delay in robots.txt don’t work for Google
- Noindex: Google used to read “Noindex: /old/” in robots.txt unofficially, but it ended support on 1 September 2019. It is now ignored. Use a meta tag on the page or X-Robots-Tag instead.
- Crawl-delay: Google doesn’t read it. Googlebot adjusts how often it crawls automatically, based on your server’s response times and errors. Bing and Yandex do read Crawl-delay.
- Host and Clean-param: used only by Yandex. Google and Bing don’t read them.
- Sitemap: Google does read it, but a bare path like “Sitemap: /sitemap.xml” is ignored. You must write a full URL starting with https://.
If Googlebot is crawling so much that your server slows down, temporarily returning a 503 or 429 status makes Google crawl less. If you keep it up for long, though, pages start dropping out of the crawl, so aim to switch back within a few days.
Common syntax mistakes and how to fix them
| Mistake | What happens | Correct version |
|---|---|---|
| User-agent：* (full-width colon) | The line isn’t read | User-agent: * |
| Disallow: https://example.com/secret/ | You must write a path, not a URL, so nothing matches | Disallow: /secret/ |
| Disallow: private/ | It doesn’t start with /, so it matches nothing | Disallow: /private/ |
| Disallow: /admin/ at the top (before any User-agent) | No crawler uses it | Add User-agent: * above it |
| User-agent: Googlebot-Images | The name doesn’t exist, so the group is never used | User-agent: Googlebot-Image |
| Noindex: /old/ | Google ignores it | noindex in the page’s meta robots tag |
| Sitemap: /sitemap.xml | Not read | Sitemap: https://example.com/sitemap.xml |
The tester’s “Commonly mistaken syntax” sample contains all of these mistakes. Load it and look at the syntax check to see how each line is read. Wildcards are explained in Allow vs Disallow precedence and what * and $ mean, and blocking AI crawlers in blocking AI crawlers with robots.txt.
FAQ
- Will blocking a page in robots.txt remove it from search results?
- Not necessarily. robots.txt only stops crawling, so a URL linked from other pages may still appear in results without a description. To remove it, unblock it and add noindex to the page.
- Can I use robots.txt and noindex together?
- If you use both on the same page, robots.txt stops Google from reading the noindex, so noindex has no effect. For pages you want out of search results, don’t block them in robots.txt; use noindex alone.
- What happens if I write Noindex in robots.txt?
- Nothing. Google ended support on 1 September 2019. Use a meta tag on the page (<meta name="robots" content="noindex">) or the X-Robots-Tag header instead.
- Should I fix “Indexed, though blocked by robots.txt”?
- If you want the page in search results, remove the robots.txt block. If you don’t, remove the block and add noindex. If you leave it as it is, the page may keep appearing in results without a description.
Tools in this article
Related articles
How to Check robots.txt: Testing for Blocks Now That the Search Console Tester Is Gone
The robots.txt Tester in Google Search Console was retired in December 2023 and replaced by the “robots.txt report”. Because the report doesn’t let you enter a URL and test whether it’s blocked, many people are left wondering where to check. This article covers where robots.txt lives and how to open it, what the Search Console report and URL Inspection tool can and can’t tell you, how to check each URL with a tester, and how changes take effect after you edit the file.
Read article →How to Write User-agent in robots.txt: What Happens When You List Both * and Googlebot
If your robots.txt has both a “User-agent: *” group and a “User-agent: Googlebot” group, Googlebot stops using the rules under “*”. A common problem is adding a Googlebot group without knowing this, only to find that pages you thought were blocked are now being crawled. This article explains how a crawler decides which group to follow, the special cases such as Googlebot-Image and AdsBot-Google, and how to combine groups and avoid common mistakes.
Read article →Blocking AI Crawlers with robots.txt: GPTBot, ClaudeBot and More
More and more site owners don’t want their writing or images used to train AI. By adding AI crawler names to robots.txt, you can ask each company’s crawler to stop fetching your pages. But AI crawlers come in three kinds, for training, for AI search and for fetching pages on a user’s request, and which ones you block makes a big difference. This article lists the main AI crawlers and what they do, shows how to write robots.txt for each goal, and explains what to watch out for.
Read article →