How does it work?
- You paste an address. The path is what gets tested.
- You pick a bot. Or type your own name.
- We read robots.txt. From the root of the site.
- We find the deciding rule. The longest matching rule wins.
How does robots.txt pick a rule?
Robots.txt is a text file that tells bots which paths they may crawl. A bot uses the group that names it, or the star group. The longest matching rule wins, and Allow beats Disallow in a tie. The idea dates from 1994, and it became the RFC 9309 standard in 2022.
| Term | What it means |
|---|---|
| User-agent | Names the bot a group of rules is for. |
| Disallow | Keeps the bot out of a path. |
| Allow | Opens a path inside a closed folder. |
| * and $ | A star matches any text. A dollar ends the path. |
Read more in the Google guide to robots.txt. Our robots.txt guide explains how to fix what you find. When you are done, run a free site audit to check the whole site.
Questions about the robots.txt tester
What if the site has no robots.txt?
Then every bot may crawl every page. We say so in the result.
Does Disallow remove a page from Google?
Not always. It stops crawling, but the address can still be listed if other pages link to it. Use a noindex tag to keep a page out. A page blocked by robots.txt cannot show Google its noindex, so allow crawling while noindex does its work.
Can I test my own bot?
Yes. Choose the custom option and type the user-agent name, using letters, digits, hyphens and underscores.
Which rule wins on a tie?
Allow wins. When two rules match with the same length, the open one is used.