How does it work?
- You type a domain. We check that it is a public site.
- We read robots.txt. One file, from the root of the site.
- We match each crawler. The most specific rule wins.
- We show the deciding line. Blocked crawlers get lines you can add.
What is an AI crawler?
An AI crawler is a bot that reads web pages for an AI product. Some collect text to train a model. Others fetch a page when a person asks a question. Your robots.txt tells each one what it may read. The robots.txt idea dates from 1994, and it became the RFC 9309 standard in 2022.
| Term | What it means |
|---|---|
| Allowed | No rule stops the crawler from reading the site. |
| Blocked | A rule keeps the crawler out of the whole site. |
| Partly blocked | Some folders are closed and others are open. |
| Applies to all | The rule is for every bot, not for this crawler by name. |
Read more in the robots.txt standard (RFC 9309). Our AI crawler guide explains how to fix what you find. When you are done, run a free site audit to check the whole site.
Questions about the AI crawler checker
Should I block AI crawlers?
It is your choice. Blocking training crawlers keeps your text out of models. Blocking the ones that fetch pages for answers can hide you from AI search.
Does robots.txt stop every bot?
No. It is a request, and well-behaved bots follow it. Tools that fetch a page for one person may ignore it.
What if there is no robots.txt?
Then every crawler is allowed. We show that case with a note, so you can add a file if you want to set rules.
Do you store my result?
Only briefly. We keep a copy for 1 hour so a repeat check is fast, and we count checks per IP address to stop abuse.