How does it work?
- You type a domain. We check that it is a public site.
- We read its sitemap. If it has none, we use the links on the home page.
- We read up to 30 pages. We take the title and description of each, 5 pages at a time.
- We write the file. Pages are grouped by section, in the llms.txt format.
Each page gets 6 seconds, and the whole read stops after 25 seconds. Pages that fail are left out. The sitemap follows the sitemaps.org protocol, so most sites have one.
What goes into the file?
The file starts with the site name and a one line summary. Then come sections of links, each with a short note. Here is a small sample.
# Acme Outdoor > Tents, packs and camp gear. ## Pages - [Home](https://acme.example/): Gear for the outdoors - [About](https://acme.example/about): Who we are ## Blog - [Pick a tent](https://acme.example/blog/tent): A short guide
The format comes from the llms.txt proposal. Our llms.txt guide explains each part. When the file is live, run a free site audit to check it.
Questions about llms.txt
What is llms.txt?
llms.txt is a plain Markdown file at the root of a site, proposed in 2024. It tells AI assistants what the site is about and which pages matter most.
Does Google use it?
Google has said it does not use llms.txt for Search, but other AI assistants and many AI agents do read it. It costs nothing, so it is still worth adding.
Where do I put the file?
Save it as llms.txt at the root of your site, so it opens at https://yourdomain/llms.txt. It should load with a 200 status as plain text.
Why only 30 pages?
A good llms.txt is a short map, not a full index, and with 30 pages an assistant can read it in one go. You can add more links by hand.
Is anything stored?
We keep each complete file for 24 hours, with only the public titles, descriptions and links of that site, so the next request is instant. To stop abuse, we count requests per IP address for 1 hour.