How to publish an ai.txt policy file for AI usage
An ai.txt file at /.well-known/ai.txt states your policy for how AI systems may use the site. It is optional and not universally supported, but some crawlers and agents read it. Create a plain text file with User-agent and Allow or Disallow lines, serve it as text/plain, and mention it in llms.txt so agents can find it.
By ratemysite.ai editorial team. Updated .
How do you fix it?
- Decide your policy: which kinds of content (text, images, code) AI systems may use for training or retrieval.
- Create a plain text file with User-agent and Allow or Disallow lines, mirroring your robots.txt decisions.
- Publish it at https://yourdomain/.well-known/ai.txt with the text/plain content type and a 200 status.
- Mention the file in your llms.txt and in your terms of use.
- Re-run the audit and confirm the ai.txt check passes.
What is ai.txt?
ai.txt is a plain text file, placed under the /.well-known/ path, that tells AI systems what they may do with a site's content. It borrows the style of robots.txt, with User-agent groups and permission lines, and covers uses such as training and retrieval. There is no single official standard yet, so support varies between tools. Think of it as a low cost, public statement of your preferences that agents can read.
How should I use it with the other files?
Use each file for its job. Robots.txt controls which crawlers may fetch pages. Llms.txt tells assistants which pages to read first. Ai.txt states how content may be used. Keep all 3 consistent, publish them over HTTPS, and review them whenever your policy changes, for example after you decide to license content. Because adoption is uneven, do not rely on ai.txt alone to block anything; use robots.txt rules and server controls for enforcement. Check the file once a quarter, since new AI crawlers appear several times a year.
How much does this check matter?
- This check is worth about 1.2% of the overall score.
- A finished report is reused for 24 hours before you can re-analyze it.
- Each page is fetched with a timeout of 10 seconds, and HTML larger than 2 MB is not audited.
What else do people ask?
- Do AI companies have to obey ai.txt?
- No. It is a voluntary signal. Robots.txt rules for named AI crawlers are the most widely respected control today.
- Should ai.txt match robots.txt?
- Keep them consistent. If robots.txt allows GPTBot but ai.txt disallows AI use, you send conflicting signals and neither will be followed reliably.