Most robots.txt files were written once and never read again. Since answer engines started crawling, that file quietly decides whether your business appears in AI answers at all. It is worth five minutes.
What robots.txt actually does
It is a plain text file at the root of your domain listing which crawlers may request which paths. It is a request, not a lock. Well behaved crawlers honour it. It is not a security measure, and it should never be used to hide private pages, because the file itself is public and effectively lists the paths you care about.
The crawlers worth knowing
They fall into three groups, and the difference matters:
- Search indexers. Googlebot and Bingbot. Blocking these removes you from search.
- Answer engine fetchers. OAI-SearchBot, PerplexityBot and similar. These retrieve pages to answer a live question and cite the source. Blocking them removes you from those answers.
- Training collectors. GPTBot, ClaudeBot, Google-Extended, CCBot. These gather text that may be used to train models. Blocking them does not remove you from search or, in most cases, from cited answers.
The common mistake is treating all three as one decision. Plenty of businesses have blocked the fetchers while meaning to block only the trainers, and wondered why they vanished from AI results.
A sensible default for a service business
If you want to be found, allow everything and spend your effort on the pages themselves:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap-index.xml
If you would rather your words were not used for training but still want to be cited, block the collectors by name and leave the fetchers alone. Keep a comment in the file saying why, so the next person does not undo it.
What usually goes wrong
Staging rules reaching production. A Disallow: / written to keep a staging site out of search gets deployed with the real site. Everything disappears within a fortnight. Check the live file after every migration.
Blocking assets. Disallowing /assets or /_astro stops crawlers rendering the page properly. They need the CSS and JavaScript to judge layout and mobile friendliness.
Using robots.txt for noindex. Blocking a page stops it being crawled, which means the crawler never sees your noindex tag, which means the page can linger in results. If you want a page out of the index, let it be crawled and serve a noindex header or meta tag.
Forgetting the sitemap line. It costs nothing and helps every crawler find everything.
llms.txt
A newer convention: a markdown file at /llms.txt that points to your most useful pages in plain language, for models reading your site. Support is patchy and it is not a ranking factor. It takes ten minutes, it cannot hurt, and it is a reasonable place to state what you do and who you serve.
Check it after you change it
Google Search Console has a robots.txt report that shows what Googlebot fetched and whether a given URL is allowed. For the rest, request the file and read it. The most common failure is believing the file says something it does not.