# Which AI crawlers read your site, and who blocks them?

mttr checks the eight crawler names that matter most for AI answers, from OpenAI's GPTBot to Anthropic's ClaudeBot and Perplexity's PerplexityBot. Of the 316,711 sites mttr has read, 4.6% shut out at least one of them by name in robots.txt, and 0.2% shut out every crawler.

By mttr. Published October 16, 2026. Updated October 27, 2026. About 4 minutes to read.

At a glance:

- 4.6%: Of sites shut out at least one AI crawler by name. Over the 316,711 sites whose robots.txt was asked, as of October 27, 2026.
- 0.2%: Of sites shut out every crawler, AI or not. 713 sites.
- 13,600: Sites shut out GPTBot by name, more than any other. 4.3% of the sites asked.
- 16%: Of sites have no robots.txt to read, which blocks nothing. Over the 316,711 sites asked.

## Which AI crawlers are there?

They come in three kinds, and the difference matters more than the names. Some collect pages to train models. Some index pages so an assistant can search them. Some fetch one page because a person asked about it just now. Blocking the first keeps your words out of training; blocking the second keeps you out of the answers.

| Token in robots.txt | Run by | What it is for | Obeys robots.txt |
| --- | --- | --- | --- |
| `GPTBot` | OpenAI | Collects pages that may train OpenAI's models | Yes |
| `ClaudeBot` | Anthropic | Collects pages that may train Anthropic's models | Yes |
| `Google-Extended` | Google | Not a crawler: a token that says whether pages Google fetched may train and ground Gemini | Yes, as a token |
| `PerplexityBot` | Perplexity | Indexes pages for Perplexity's answers; not for training, Perplexity says | Yes |
| `ChatGPT-User` | OpenAI | Fetches a page when a person asks ChatGPT to | OpenAI says its rules may not apply |
| `Claude-SearchBot` | Anthropic | Indexes pages for search in Claude | Yes |
| `OAI-SearchBot` | OpenAI | Indexes pages for search in ChatGPT | Yes |
| `bingbot` | Microsoft | Bing's search crawler, whose index Copilot answers from | Yes |

Each operator says so on its own page, linked under Sources. Perplexity also runs a user fetcher, Perplexity-User, which it says generally ignores robots.txt, and Anthropic runs Claude-User for the same job. A rule aimed at a user fetcher is a request, not a lock.

## How many sites block each AI crawler?

Few, and far fewer than the noise about it suggests. 4.6% of the 316,711 sites mttr has read name at least one AI crawler and shut it out; 0.2% shut out every crawler, AI or not. GPTBot is shut out by name on 13,600 sites and OAI-SearchBot on 2,998: most sites that block are opting out of training, and many are still letting themselves be found.

Sites whose robots.txt shuts out each crawler by name:

| Crawler | Sites blocking it by name | Sites | With sites closed to all |
| --- | ---: | ---: | ---: |
| GPTBot (OpenAI) | 4.3% | 13,600 | 4.5% |
| ClaudeBot (Anthropic) | 3.6% | 11,445 | 3.8% |
| Google-Extended (Google) | 3.1% | 9,662 | 3.3% |
| PerplexityBot (Perplexity) | 1.9% | 5,964 | 2.1% |
| ChatGPT-User (OpenAI) | 1.5% | 4,880 | 1.8% |
| Claude-SearchBot (Anthropic) | 1.0% | 3,297 | 1.3% |
| OAI-SearchBot (OpenAI) | 0.9% | 2,998 | 1.2% |
| bingbot (Microsoft) | 0.1% | 452 | 0.4% |

_Counted from the 316,711 sites mttr has read, each at its latest read, as of October 27, 2026. A crawler counts as blocked when the robots.txt rules that apply to it disallow the home page; a rule that closes only part of a site is not counted. The last column adds the 713 sites that shut out every crawler. 16% of these sites have no robots.txt to read, which blocks nothing._

## What does robots.txt control, and what does it not?

robots.txt asks crawlers not to fetch paths, one name at a time. The standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), says plainly that its rules are not a form of access authorization. It is honoured by crawlers that choose to honour it, and the large ones say they do.

- It does control whether a well behaved crawler fetches a page, for each token by name.
- It does not take back pages already collected, or a model already trained.
- It does not bind a fetch a person asked for. OpenAI and Perplexity both say their user fetchers may not follow it.
- It does not stop a firewall or a CDN from refusing a crawler, which robots.txt never shows. A site can allow GPTBot in robots.txt and still answer it with a 403.
- Google-Extended does not take a site out of Google Search, and Google says the AI features in Search follow the controls Search already has, such as `nosnippet` and `noindex`.

## Should you block AI crawlers?

Decide training and search separately. If you want an assistant to cite you, let the search and user crawlers in: blocking GPTBot does not take you out of search in ChatGPT, which OpenAI runs on OAI-SearchBot. If you want your words out of training, block the training crawlers by name and say nothing about the rest.

Choose who gets in. The file and what it means follow.

- `GPTBot` (OpenAI), collect pages to train models: shut out by name on 4.3% of sites
- `ClaudeBot` (Anthropic), collect pages to train models: shut out by name on 3.6% of sites
- `Google-Extended` (Google), collect pages to train models: shut out by name on 3.1% of sites
- `OAI-SearchBot` (OpenAI), index pages for assistants' search: shut out by name on 0.9% of sites
- `Claude-SearchBot` (Anthropic), index pages for assistants' search: shut out by name on 1.0% of sites
- `PerplexityBot` (Perplexity), index pages for assistants' search: shut out by name on 1.9% of sites
- `bingbot` (Microsoft), index pages for assistants' search: shut out by name on 0.1% of sites
- `ChatGPT-User` (OpenAI), fetch a page a person asked about: shut out by name on 1.5% of sites

The file that keeps training crawlers out and lets the rest in:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml
```

_Each share is of the sites whose robots.txt shuts that crawler out by name. Counted from the 316,711 sites mttr has read, each at its latest read, as of October 27, 2026._

On Cloudflare, a setting called Block AI bots writes rules like these for you, for every AI crawler at once. Check what it wrote before you leave it on.

## How do you check your own robots.txt?

The free [AI crawler checker](https://mttr.si/ai-crawler-checker) reads a site's robots.txt and says which of these crawlers it lets in, and whether there is an llms.txt. A full read with mttr holds the site to the rule [Let AI crawlers in](https://mttr.si/rulebook) among the rest.

See who your robots.txt lets in: https://mttr.si/ai-crawler-checker?url=<address>. Type your domain. The checker reads its robots.txt as each of these crawlers would and says who is let in, and whether there is an llms.txt.

Write the file with your agent. Hand this to your agent:

> Use mttr to read yoursite.com and tell me which AI crawlers its robots.txt shuts out. Then write me a robots.txt that keeps GPTBot, ClaudeBot and Google-Extended out of training and lets the search and user crawlers in, keeping every rule the file has today for other crawlers. Show me the file before you change it.

The calls it makes: `mttr_brief({ site: "yoursite.com" })`, `mttr_rescan({ site: "yoursite.com" })`.

### Settle your robots.txt

- Decide training and search separately.
- Name each training crawler you want out, and only those.
- Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot in if you want to be cited.
- Keep a `Sitemap` line with the full address of your sitemap.
- Check that your firewall or CDN does not refuse the crawlers robots.txt lets in.
- Read the file again after any setting that writes it for you.

`mttr_brief({ site: "yoursite.com" })`: Every fix, the robots.txt findings among them, as Markdown for an agent

## Sources

- [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots): GPTBot, OAI-SearchBot and ChatGPT-User
- [Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518): ClaudeBot, Claude-SearchBot and Claude-User
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots): PerplexityBot and Perplexity-User
- [Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers): Google-Extended and what it does not affect
- [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features): Google Search Central
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309): The standard for robots.txt, 2022

## How the figures were counted

Every figure here is counted from the 316,711 sites mttr has read, each at its latest read, with drafts left out, as of October 27, 2026. A share of pages counts only pages that were read. A score or a rule passed says pages are well formed. It does not say an assistant cites the site.