# robots.txt and AI crawlers: what to allow | Plyo

> Which bots are asking for your pages, what each one does with them, and how to write a robots.txt that reflects a decision rather than an accident.

Source: https://plyo.studio/blog/robots-txt-and-ai-crawlers

1.  [Home](/)
2.  [Blog](/blog)
3.  robots.txt and AI crawlers: what to allow

# robots.txt and AI crawlers: what to allow

Which bots are asking for your pages, what each one does with them, and how to write a robots.txt that reflects a decision rather than an accident.

Plyo Studio 14 May 2026 7 min read

On this page

[What robots.txt actually does](#what-robotstxt-actually-does)[The crawlers worth knowing](#the-crawlers-worth-knowing)[A sensible default for a service business](#a-sensible-default-for-a-service-business)[What usually goes wrong](#what-usually-goes-wrong)[llms.txt](#llmstxt)[Check it after you change it](#check-it-after-you-change-it) [Common questions](#common-questions)

Related service

[Website speed & SEO audit](/services/website-audit)

Most robots.txt files were written once and never read again. Since answer engines started crawling, that file quietly decides whether your business appears in AI answers at all. It is worth five minutes.

## What robots.txt actually does

It is a plain text file at the root of your domain listing which crawlers may request which paths. It is a request, not a lock. Well behaved crawlers honour it. It is not a security measure, and it should never be used to hide private pages, because the file itself is public and effectively lists the paths you care about.

## The crawlers worth knowing

They fall into three groups, and the difference matters:

-   **Search indexers.** Googlebot and Bingbot. Blocking these removes you from search.
-   **Answer engine fetchers.** OAI-SearchBot, PerplexityBot and similar. These retrieve pages to answer a live question and cite the source. Blocking them removes you from those answers.
-   **Training collectors.** GPTBot, ClaudeBot, Google-Extended, CCBot. These gather text that may be used to train models. Blocking them does not remove you from search or, in most cases, from cited answers.

The common mistake is treating all three as one decision. Plenty of businesses have blocked the fetchers while meaning to block only the trainers, and wondered why they vanished from AI results.

## A sensible default for a service business

If you want to be found, allow everything and spend your effort on the pages themselves:

```
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap-index.xml
```

If you would rather your words were not used for training but still want to be cited, block the collectors by name and leave the fetchers alone. Keep a comment in the file saying why, so the next person does not undo it.

## What usually goes wrong

**Staging rules reaching production.** A `Disallow: /` written to keep a staging site out of search gets deployed with the real site. Everything disappears within a fortnight. Check the live file after every migration.

**Blocking assets.** Disallowing /assets or /\_astro stops crawlers rendering the page properly. They need the CSS and JavaScript to judge layout and mobile friendliness.

**Using robots.txt for noindex.** Blocking a page stops it being crawled, which means the crawler never sees your noindex tag, which means the page can linger in results. If you want a page out of the index, let it be crawled and serve a noindex header or meta tag.

**Forgetting the sitemap line.** It costs nothing and helps every crawler find everything.

## llms.txt

A newer convention: a markdown file at /llms.txt that points to your most useful pages in plain language, for models reading your site. Support is patchy and it is not a ranking factor. It takes ten minutes, it cannot hurt, and it is a reasonable place to state what you do and who you serve.

## Check it after you change it

Google Search Console has a robots.txt report that shows what Googlebot fetched and whether a given URL is allowed. For the rest, request the file and read it. The most common failure is believing the file says something it does not.

## Keep reading.

[All posts](/blog)

-   [
    
    Search & AEO What to do when your website traffic drops 8 min read](/blog/when-website-traffic-drops)
-   [
    
    Search & AEO Google Business Profile and your website 7 min read](/blog/google-business-profile-and-your-website)
-   [
    
    Search & AEO How to get your business quoted by ChatGPT 8 min read](/blog/how-to-get-quoted-by-chatgpt)

Updated 14 May 2026

FAQ

## Common questions

[Ask a question](/contact)

### Does blocking GPTBot remove me from ChatGPT answers?

Usually not. GPTBot is the training collector. Live answers typically use a separate fetcher, so blocking GPTBot alone does not hide you from citations.

### Should a small business block AI crawlers?

Rarely. Visibility is worth more than the content itself for most service businesses. Publishers who sell access to their archive are the exception.

### Does robots.txt keep private pages private?

No. It is a public file and it is advisory. Anything genuinely private needs a login.

### Where does the file go?

At the root of the domain, at /robots.txt. One file per domain, and subdomains need their own.

### How often should I check it?

After every migration, redesign or hosting change, and once a quarter otherwise.

About Plyo Studio

## Custom Astro websites. Built on Cloudflare.

Plyo Studio is an independent web studio that designs and builds fast, search-ready websites, web apps and interactive maps. Sites are custom-built in Astro and hosted on Cloudflare, so they load quickly and are easy for search engines and AI answer engines to read.

Founded by Chloe Nixon, a full stack developer and UI/UX designer from Australia, and Micky, a software engineer from New Zealand. Clients work directly with the founders from the first call to launch. Typical work: custom websites, migrations off Webflow, WordPress, Wix and Squarespace, web apps, and interactive maps and GIS. Projects are quoted at a fixed price before work starts.

[Start a project](/contact) [hello@plyo.studio](mailto:hello@plyo.studio) [About us](/about)

## Related pages

-   [Blog Read practical notes on designing, building and moving websites.](/blog)
-   [Website speed & SEO audit A clear report on what is slowing your site down and costing you rankings.](/services/website-audit)

Start a conversation

## Ready for a website that works harder?

[Start a project](/contact) [Free website audit](/audit)
