How to Write User-agent in robots.txt: What Happens When You List Both * and Googlebot
If your robots.txt has both a “User-agent: *” group and a “User-agent: Googlebot” group, Googlebot stops using the rules under “*”. A common problem is adding a Googlebot group without knowing this, only to find that pages you thought were blocked are now being crawled. This article explains how a crawler decides which group to follow, the special cases such as Googlebot-Image and AdsBot-Google, and how to combine groups and avoid common mistakes.
What a group is: User-agent lines and the rules below them
A robots.txt file is a collection of groups. A group starts with one or more User-agent lines, followed by Allow and Disallow rules. A new group begins at the next User-agent line that comes after some rules.
| Syntax | Meaning |
|---|---|
| User-agent: * / Disallow: /tmp/ | Block /tmp/ for every crawler that has no group of its own |
| User-agent: GPTBot / User-agent: ClaudeBot / Disallow: / | Block the whole site for both GPTBot and ClaudeBot (one group) |
| User-agent: Googlebot / Disallow: | Block nothing for Googlebot (an empty Disallow allows everything) |
Blank lines between groups are only there for readability; the file works without them. Sitemap lines don’t belong to any group and apply to every crawler wherever you put them. Be careful: Allow and Disallow lines written before the first User-agent line are not used by any crawler.
A crawler follows only one group
A crawler looks through robots.txt for a group with its own name and, if it finds one, uses only that group’s rules. It falls back to the “User-agent: *” group only when there is no group with its name. Rules from two groups are never added together.
- 1
Example: when both * and Googlebot exist
Suppose “User-agent: *” has Disallow: /private/ and Disallow: /tmp/, and “User-agent: Googlebot” has only Disallow: /no-google/.
- 2
How Googlebot reads it
Googlebot uses only its own group, so /no-google/ is blocked while /private/ and /tmp/ are allowed. Pages you meant to block with * end up being crawled.
- 3
How Bingbot reads it
There’s no group named Bingbot, so it uses “*”. /private/ and /tmp/ are blocked, and /no-google/ is allowed.
- 4
How to fix it
Add Disallow: /private/ and Disallow: /tmp/ to the Googlebot group as well. If Googlebot doesn’t need special treatment, the simplest fix is to delete the Googlebot group and keep only “*”.
Special handling for Google’s crawlers
Google has several crawlers for different purposes, and each picks its group in a slightly different way.
| Crawler | Group it uses |
|---|---|
| Googlebot | The Googlebot group → otherwise * |
| Googlebot-Image / -Video / -News | Its own group → otherwise Googlebot → otherwise * |
| AdsBot-Google | Only its own group (ignores *) |
| Mediapartners-Google (AdSense) | Only its own group (ignores *) |
| Google-Extended | Its own group (only controls use for training and answers in Gemini and other AI) |
| Applebot (Apple) | The Applebot group → otherwise follows the Googlebot group |
- If you have an image folder you don’t want in image search, block it with “User-agent: Googlebot-Image”. Regular Google Search crawling isn’t affected.
- Writing “Disallow: /” under “User-agent: *” does not stop AdsBot-Google. Name it only if you want to stop the quality checks on ad landing pages (normally there’s no need).
- Blocking the AdSense crawler makes ads less relevant to your content, so sites that use AdSense usually leave it alone.
- If you write a name that doesn’t exist, such as “Googlebot-Images” (with an s), no crawler will ever use that group. The tester also flags misspelled names.
Combining several crawlers, and when the same name appears twice
To apply the same rules to several crawlers, write the User-agent lines one after another and the rules only once below them. This is the usual way to block a batch of AI crawlers.
If a group with the same name appears twice in robots.txt, Google merges the rules of both groups. Other crawlers may not do the same, though, and it’s confusing when you come back to the file later, so it’s best to keep them in one place.
- Write only the name as the User-agent value. If you write “Googlebot/2.1” or a whole browser UA string, only the name at the start is used.
- Names are not case-sensitive.
- A crawler whose group has no rules at all is allowed everything, and it doesn’t use the “*” rules either.
- Use a half-width colon “:”. With a full-width colon “：” the line isn’t read.
AI crawler names and how to block them are covered in blocking AI crawlers with robots.txt, and the precedence between rules in Allow vs Disallow precedence.
FAQ
- What happens if I write both User-agent: * and User-agent: Googlebot?
- Googlebot uses only the rules in its own group and ignores the rules in the “*” group. If you want Googlebot to stay out of the paths blocked under “*”, write the same Disallow lines in the Googlebot group too.
- Can I block only Googlebot-Image?
- Yes. Put Disallow lines in a “User-agent: Googlebot-Image” group and only the image search crawler is blocked. If there’s no group with its name, Googlebot-Image follows the Googlebot group.
- Does User-agent: * also block AdsBot-Google?
- No. AdsBot-Google and AdSense’s Mediapartners-Google ignore the “*” group, so you have to name them if you want to block them.
- How do I apply the same rules to several User-agents?
- Write the User-agent lines one after another and the rules only once below them. For example, two lines “User-agent: GPTBot” and “User-agent: ClaudeBot” followed by “Disallow: /” apply the same rule to both.
Tools in this article
Related articles
How to Check robots.txt: Testing for Blocks Now That the Search Console Tester Is Gone
The robots.txt Tester in Google Search Console was retired in December 2023 and replaced by the “robots.txt report”. Because the report doesn’t let you enter a URL and test whether it’s blocked, many people are left wondering where to check. This article covers where robots.txt lives and how to open it, what the Search Console report and URL Inspection tool can and can’t tell you, how to check each URL with a tester, and how changes take effect after you edit the file.
Read article →Blocking AI Crawlers with robots.txt: GPTBot, ClaudeBot and More
More and more site owners don’t want their writing or images used to train AI. By adding AI crawler names to robots.txt, you can ask each company’s crawler to stop fetching your pages. But AI crawlers come in three kinds, for training, for AI search and for fetching pages on a user’s request, and which ones you block makes a big difference. This article lists the main AI crawlers and what they do, shows how to write robots.txt for each goal, and explains what to watch out for.
Read article →robots.txt vs noindex: The Right Way to Remove a Page from Search Results
“I blocked it in robots.txt, but it still shows up in search.” “Search Console says ‘Indexed, though blocked by robots.txt’.” Most of these problems come from mixing up what robots.txt and noindex do. This article explains the difference between crawling and indexing, the correct steps to remove a page from search results, why Noindex and Crawl-delay in robots.txt don’t work, and how to deal with common Search Console messages and syntax mistakes.
Read article →