How to remove noai and noindex X-Robots-Tag headers
An X-Robots-Tag header with noai, noimageai or a per-bot noindex tells AI engines to skip the page. Unless you intend that, inspect the response headers, find where each directive is set, remove the AI directives at the server or CDN, and re-audit the page.
By ratemysite.ai editorial team. Updated .
How do you fix it?
- Run curl -I https://yourdomain/page and read every X-Robots-Tag header in the response.
- Look for noai, noimageai, or bot-specific values such as "gptbot: noindex" or "googlebot: noindex".
- Find where the header is set: web server config, CDN transform rule, security plugin or framework middleware.
- Remove the directives you did not intend, and keep them only for pages you truly want hidden.
- Re-run the audit and confirm the X-Robots AI check passes.
What is the X-Robots-Tag?
The X-Robots-Tag is an HTTP response header that carries indexing instructions, like the robots meta tag but applied at the server level, so it also works for PDFs and images. Values such as noindex tell engines to drop a page, and values such as noai or noimageai ask AI systems not to use the content. Because it lives in the headers, it is invisible in the page source and often forgotten for years.
Where do unwanted headers come from?
Common sources are a security or privacy plugin with a block AI option, a CDN rule added during a launch, and a staging config that was copied to production. Check headers on several URL types, such as the home page, a post, an image and a PDF, because rules often target one file type. Search your server and CDN config for the string X-Robots-Tag. A common pattern is a staging config with an X-Robots-Tag line that blocks AI crawlers and gets copied to 1 production server unchanged. After removing it, request the page again with curl and confirm the header is gone from every variant, including www, non-www and any CDN cached copy.
How much does this check matter?
- This check is worth about 2.4% of the overall score.
- A finished report is reused for 24 hours before you can re-analyze it.
- Each page is fetched with a timeout of 10 seconds, and HTML larger than 2 MB is not audited.
What else do people ask?
- Is noai an official standard?
- No. It is a convention some platforms and tools honor. Its support varies, but a header can still signal to crawlers that read it.
- Can I block AI on a few pages only?
- Yes. Send the header only for those paths, and leave the rest of the site open.