How to create a robots.txt that does not block crawlers
A robots.txt file at the site root tells crawlers which paths they may fetch. It should exist, allow the home page for Googlebot, and name your sitemap. A missing file is tolerated, but a stray Disallow: / hides the whole site from search.
By ratemysite.ai editorial team. Updated .
How do you fix it?
- Create a plain text file named robots.txt and serve it at https://yourdomain/robots.txt.
- Add "User-agent: *" followed by "Allow: /" and only the Disallow lines you really need, such as admin paths.
- Add a line "Sitemap: https://yourdomain/sitemap.xml" at the end.
- Open the file in a browser and test your key URLs with the Search Console robots.txt report.
- Re-run the audit and confirm the robots.txt check passes.
What is robots.txt?
Robots.txt is a plain text file at the root of a website that gives crawlers rules about which URLs they may request. Every major search engine and most AI crawlers read it before fetching pages. The rules are public, so never list secret paths in it. A single typo can be costly: Disallow: / under User-agent: * blocks the entire site, and it is one of the most common leftovers when a staging site goes live.
What should a safe robots.txt contain?
For most sites, 3 lines are enough: a User-agent: * group, an Allow: / rule, and a Sitemap line. Add Disallow lines only for areas with no search value, such as internal search results, cart pages or admin folders. Keep rules for AI crawlers in their own groups so each decision is clear. Re-check the file after every migration or redesign, because platforms sometimes overwrite it with a restrictive default.
How much does this check matter?
- This check is worth about 1.9% of the overall score.
- A finished report is reused for 24 hours before you can re-analyze it.
- Each page is fetched with a timeout of 10 seconds, and HTML larger than 2 MB is not audited.
What else do people ask?
- Does robots.txt hide pages from Google?
- Not reliably. It blocks crawling, but a blocked URL can still be indexed if other sites link to it. Use a noindex tag to keep a page out of results.
- What happens if robots.txt returns an error?
- A 404 is treated as no rules, so everything is crawlable. A server error such as 500 can pause crawling, so keep the file reliable.