Free audit
Menu
Free tool

Free AI crawler checker

Type a domain to see which AI crawlers your robots.txt lets in. Each crawler is listed as allowed, blocked or partly blocked, with the rule that decides it. Blocked ones come with a fix you can copy.

  • 16 AI crawlers
  • 20 checks per hour

By . Updated .

Does my site let AI crawlers in?

Enter your domain below. We read your robots.txt and match it against each AI crawler. The home page is the only other page we load.

The button turns on after a quick human check, which starts when you click the field.

The human check is Cloudflare Turnstile. Each IP address can make 20 checks in 60 minutes, and a result is kept for 60 minutes.

How does it work?

  • You type a domain. We check that it is a public site.
  • We read robots.txt. One file, from the root of the site.
  • We match each crawler. The most specific rule wins.
  • We show the deciding line. Blocked crawlers get lines you can add.

What is an AI crawler?

An AI crawler is a bot that reads web pages for an AI product. Some collect text to train a model. Others fetch a page when a person asks a question. Your robots.txt tells each one what it may read. The robots.txt idea dates from 1994, and it became the RFC 9309 standard in 2022.

TermWhat it means
AllowedNo rule stops the crawler from reading the site.
BlockedA rule keeps the crawler out of the whole site.
Partly blockedSome folders are closed and others are open.
Applies to allThe rule is for every bot, not for this crawler by name.

Read more in the robots.txt standard (RFC 9309). Our AI crawler guide explains how to fix what you find. When you are done, run a free site audit to check the whole site.

Questions about the AI crawler checker

Should I block AI crawlers?

It is your choice. Blocking training crawlers keeps your text out of models. Blocking the ones that fetch pages for answers can hide you from AI search.

Does robots.txt stop every bot?

No. It is a request, and well-behaved bots follow it. Tools that fetch a page for one person may ignore it.

What if there is no robots.txt?

Then every crawler is allowed. We show that case with a note, so you can add a file if you want to set rules.

Do you store my result?

Only briefly. We keep a copy for 1 hour so a repeat check is fast, and we count checks per IP address to stop abuse.

See all 49 checks on your site

Run a free audit