Custom Astro websites. Built on Cloudflare.

Explore Plyo

robots.txt and AI crawlers: what to allow

Which bots are asking for your pages, what each one does with them, and how to write a robots.txt that reflects a decision rather than an accident.

A line drawing of a rules file with one gate open and one closed on a pale blue cover

Most robots.txt files were written once and never read again. Since answer engines started crawling, that file quietly decides whether your business appears in AI answers at all. It is worth five minutes.

What robots.txt actually does

It is a plain text file at the root of your domain listing which crawlers may request which paths. It is a request, not a lock. Well behaved crawlers honour it. It is not a security measure, and it should never be used to hide private pages, because the file itself is public and effectively lists the paths you care about.

The crawlers worth knowing

They fall into three groups, and the difference matters:

  • Search indexers. Googlebot and Bingbot. Blocking these removes you from search.
  • Answer engine fetchers. OAI-SearchBot, PerplexityBot and similar. These retrieve pages to answer a live question and cite the source. Blocking them removes you from those answers.
  • Training collectors. GPTBot, ClaudeBot, Google-Extended, CCBot. These gather text that may be used to train models. Blocking them does not remove you from search or, in most cases, from cited answers.

The common mistake is treating all three as one decision. Plenty of businesses have blocked the fetchers while meaning to block only the trainers, and wondered why they vanished from AI results.

A sensible default for a service business

If you want to be found, allow everything and spend your effort on the pages themselves:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap-index.xml

If you would rather your words were not used for training but still want to be cited, block the collectors by name and leave the fetchers alone. Keep a comment in the file saying why, so the next person does not undo it.

What usually goes wrong

Staging rules reaching production. A Disallow: / written to keep a staging site out of search gets deployed with the real site. Everything disappears within a fortnight. Check the live file after every migration.

Blocking assets. Disallowing /assets or /_astro stops crawlers rendering the page properly. They need the CSS and JavaScript to judge layout and mobile friendliness.

Using robots.txt for noindex. Blocking a page stops it being crawled, which means the crawler never sees your noindex tag, which means the page can linger in results. If you want a page out of the index, let it be crawled and serve a noindex header or meta tag.

Forgetting the sitemap line. It costs nothing and helps every crawler find everything.

llms.txt

A newer convention: a markdown file at /llms.txt that points to your most useful pages in plain language, for models reading your site. Support is patchy and it is not a ranking factor. It takes ten minutes, it cannot hurt, and it is a reasonable place to state what you do and who you serve.

Check it after you change it

Google Search Console has a robots.txt report that shows what Googlebot fetched and whether a given URL is allowed. For the rest, request the file and read it. The most common failure is believing the file says something it does not.

Updated

FAQ

Common questions

Ask a question

Does blocking GPTBot remove me from ChatGPT answers?

Usually not. GPTBot is the training collector. Live answers typically use a separate fetcher, so blocking GPTBot alone does not hide you from citations.

Should a small business block AI crawlers?

Rarely. Visibility is worth more than the content itself for most service businesses. Publishers who sell access to their archive are the exception.

Does robots.txt keep private pages private?

No. It is a public file and it is advisory. Anything genuinely private needs a login.

Where does the file go?

At the root of the domain, at /robots.txt. One file per domain, and subdomains need their own.

How often should I check it?

After every migration, redesign or hosting change, and once a quarter otherwise.

About Plyo Studio

Custom Astro websites. Built on Cloudflare.

Plyo Studio is an independent web studio that designs and builds fast, search-ready websites, web apps and interactive maps. Sites are custom-built in Astro and hosted on Cloudflare, so they load quickly and are easy for search engines and AI answer engines to read.

Founded by Chloe Nixon, a full stack developer and UI/UX designer from Australia, and Micky, a software engineer from New Zealand. Clients work directly with the founders from the first call to launch. Typical work: custom websites, migrations off Webflow, WordPress, Wix and Squarespace, web apps, and interactive maps and GIS. Projects are quoted at a fixed price before work starts.