Free audit
Menu
Free tool

Free robots.txt tester

Paste a page address and pick a bot. We read the robots.txt of the site and tell you whether that bot may crawl the page, with the exact rule that decides.

  • 8 bots or your own
  • 20 checks per hour

By . Updated .

Can this bot crawl my page?

Paste the full address of a page and choose a bot. We fetch robots.txt from the site and test the path against it.

The button turns on after a quick human check, which starts when you click the field.

The human check is Cloudflare Turnstile. Each IP address can make 20 checks in 60 minutes, and a result is kept for 60 minutes.

How does it work?

  • You paste an address. The path is what gets tested.
  • You pick a bot. Or type your own name.
  • We read robots.txt. From the root of the site.
  • We find the deciding rule. The longest matching rule wins.

How does robots.txt pick a rule?

Robots.txt is a text file that tells bots which paths they may crawl. A bot uses the group that names it, or the star group. The longest matching rule wins, and Allow beats Disallow in a tie. The idea dates from 1994, and it became the RFC 9309 standard in 2022.

TermWhat it means
User-agentNames the bot a group of rules is for.
DisallowKeeps the bot out of a path.
AllowOpens a path inside a closed folder.
* and $A star matches any text. A dollar ends the path.

Read more in the Google guide to robots.txt. Our robots.txt guide explains how to fix what you find. When you are done, run a free site audit to check the whole site.

Questions about the robots.txt tester

What if the site has no robots.txt?

Then every bot may crawl every page. We say so in the result.

Does Disallow remove a page from Google?

Not always. It stops crawling, but the address can still be listed if other pages link to it. Use a noindex tag to keep a page out. A page blocked by robots.txt cannot show Google its noindex, so allow crawling while noindex does its work.

Can I test my own bot?

Yes. Choose the custom option and type the user-agent name, using letters, digits, hyphens and underscores.

Which rule wins on a tie?

Allow wins. When two rules match with the same length, the open one is used.

See all 49 checks on your site

Run a free audit