robots.txt Allow vs Disallow Precedence: What * and $ Mean
When both an Allow and a Disallow rule in robots.txt match the same URL, which one wins? Many people assume "whichever is written first", but that's not how Google reads it. This article uses examples to explain the "longest matching rule wins" approach Google uses, what the * wildcard and a trailing $ mean, the difference between "/fish" and "/fish/", and how case and non-ASCII URLs are handled.
The basic rule: the longest matching rule wins
Google gathers every Allow and Disallow rule that matches the URL path and uses the one whose pattern has the most characters. The idea is that a longer pattern is a more specific instruction. The order the rules are written in doesn't matter. This way of reading the file is also written into the robots.txt standard (RFC 9309).
| robots.txt | URL | Result | Why |
|---|---|---|---|
| Disallow: /blog/ / Allow: /blog/public/ | /blog/public/a.html | Allowed | The Allow (13 characters) is longer than the Disallow (6 characters) |
| Disallow: /blog/ / Allow: /blog/public/ | /blog/secret.html | Blocked | Only Disallow: /blog/ matches |
| Allow: /blog/ / Disallow: /blog/private/ | /blog/private/x | Blocked | The Disallow is longer. Having the Allow above it makes no difference |
| Disallow: / / Allow: /$ | / | Allowed | /$ is longer than / ($ counts as a character too) |
| Disallow: / / Allow: /$ | /about | Blocked | /$ matches only the home page |
Bing reads files in much the same way, but some older crawlers and tools use "the first rule that matches, reading from the top". If you want the same result for every crawler, it's safer to put Allow rules above Disallow rules.
Paths match by prefix: "/fish" vs "/fish/"
The path in an Allow or Disallow rule is compared against the start of the URL (prefix matching). So whether there's a trailing slash makes a big difference to what gets blocked.
| Pattern | Matches | Doesn't match |
|---|---|---|
| /fish | /fish, /fish.html, /fish/salmon.html, /fishheads | /Fish.asp, /catfish, /?id=fish |
| /fish/ | /fish/, /fish/salmon.html, /fish/?id=1 | /fish, /fish.html, /Fish/Salmon.asp |
| / | Every URL on the site | — |
| (empty) | Matches nothing (Disallow: on its own allows everything) | — |
Writing "Disallow: /admin" blocks not only /admin/ but also /administrator and /admin-guide.html. If you only want to block what's inside the folder, add a trailing slash: "Disallow: /admin/". This difference is also why WordPress's default "Disallow: /wp-admin/" doesn't block /wp-admin-guide/.
What "*" and "$" mean and how to use them
Google and Bing treat two symbols in patterns specially. "*" matches any string of zero or more characters, and "$" means "end of the URL", but only when it's placed at the end of the pattern.
| Pattern | Meaning | Matches | Doesn't match |
|---|---|---|---|
| /*.php | Contains .php somewhere | /index.php, /a.php?x=1, /a.php5 | /windows.PHP |
| /*.php$ | Ends in .php | /index.php, /dir/a.php | /a.php?x=1, /a.php5 |
| /*.pdf$ | Ends in .pdf | /files/manual.pdf | /files/manual.pdf?download=1 |
| /*? | Has a query string (?) | /?a=1, /items?color=red | /items |
| /*?*sort= | Query string contains sort= | /items?color=red&sort=price | /items?color=red |
| /fish* | Same as /fish (a trailing * does nothing) | /fish, /fishheads | /catfish |
- "$" only has its special meaning at the end. A $ in the middle, as in "/a$b", is treated as an ordinary character.
- "**" is the same as a single "*".
- You can't use wildcards in User-agent names, as in "*bot". The only place "*" works is the form "User-agent: *".
- Patterns start with "/" or "*". If you forget the /, as in "Disallow: private/", it won't match any URL.
Examples: keeping search results and parameter URLs out of the crawl
Site search results and URLs with sorting or filtering parameters can multiply almost endlessly depending on the combinations. They're often excluded in robots.txt to reduce wasted crawling.
- 1
Disallow: /search
Blocks /search and /search?q=shoes. Note that because it's a prefix match, URLs such as /search-tips are blocked too.
- 2
Disallow: /*?s=
Blocks WordPress site search (/?s=keyword, /blog/?s=…).
- 3
Disallow: /*?*sort=
Blocks URLs whose query string contains sort= anywhere. It doesn't block /items?color=red but does block /items?color=red&sort=price.
- 4
Allow: /search/help
Allows only the help page within the /search block. Because the Allow is longer, it's allowed regardless of the order you write them in.
Some patterns stop matching when the order of parameters changes, so putting a * in between, as in "/*?*sort=" rather than "/*?sort=", reduces gaps. List a few kinds of real URLs and check the results in a tester before publishing.
Case and non-ASCII URLs
- Paths are case-sensitive. "Disallow: /Admin/" doesn't block /admin/. With a CMS whose URLs mix in capitals, write both or match the actual URLs.
- User-agent names aren't case-sensitive. "googlebot" and "Googlebot" are the same.
- For non-ASCII paths, Google treats "Disallow: /商品/" written as-is and "Disallow: /%E5%95%86%E5%93%81/" with percent-encoding as the same thing. Save the file as UTF-8.
- The letters in percent-encoding (%E5 vs %e5) aren't case-sensitive.
Like Google, the robots.txt Tester compares non-ASCII paths and their percent-encoded forms as the same URL. It's reassuring to try it with real URLs and confirm that both ways of writing give the same result. For how groups are chosen, see How to write User-agent and how groups are chosen, and for reasons a rule you wrote has no effect, see How to check robots.txt.
FAQ
- Do rules written higher up in robots.txt take priority?
- For Google, the order doesn't matter. Of the rules that match, it uses the one with the longest pattern, and on a tie it uses Allow. If you also want to cater for crawlers that read from the top, it's safer to put Allow first.
- What does $ mean in robots.txt?
- A "$" at the end of a pattern means "end of the URL". "Disallow: /*.pdf$" blocks only URLs ending in .pdf, and doesn't block URLs followed by a query string, such as /a.pdf?download=1.
- Are Disallow: /admin and Disallow: /admin/ different?
- Yes. "/admin" is a prefix match, so it also blocks /administrator and /admin.html. "/admin/" blocks only what's inside the folder, and doesn't block /admin (without the slash).
- What happens if I write both Allow: / and Disallow: /?
- Both patterns are 1 character long, so Google uses Allow and everything is allowed. If you want to block the whole site, delete Allow: /.
Tools in this article
Related articles
How to Check robots.txt: Testing for Blocks Now That the Search Console Tester Is Gone
The robots.txt Tester in Google Search Console was retired in December 2023 and replaced by the “robots.txt report”. Because the report doesn’t let you enter a URL and test whether it’s blocked, many people are left wondering where to check. This article covers where robots.txt lives and how to open it, what the Search Console report and URL Inspection tool can and can’t tell you, how to check each URL with a tester, and how changes take effect after you edit the file.
Read article →How to Write User-agent in robots.txt: What Happens When You List Both * and Googlebot
If your robots.txt has both a “User-agent: *” group and a “User-agent: Googlebot” group, Googlebot stops using the rules under “*”. A common problem is adding a Googlebot group without knowing this, only to find that pages you thought were blocked are now being crawled. This article explains how a crawler decides which group to follow, the special cases such as Googlebot-Image and AdsBot-Google, and how to combine groups and avoid common mistakes.
Read article →Blocking AI Crawlers with robots.txt: GPTBot, ClaudeBot and More
More and more site owners don’t want their writing or images used to train AI. By adding AI crawler names to robots.txt, you can ask each company’s crawler to stop fetching your pages. But AI crawlers come in three kinds, for training, for AI search and for fetching pages on a user’s request, and which ones you block makes a big difference. This article lists the main AI crawlers and what they do, shows how to write robots.txt for each goal, and explains what to watch out for.
Read article →