Which AI crawlers read your site, and who blocks them?

mttr checks the eight crawler names that matter most for AI answers, from OpenAI's GPTBot to Anthropic's ClaudeBot and Perplexity's PerplexityBot. Of the 316,711 sites mttr has read, 4.6% shut out at least one of them by name in robots.txt, and 0.2% shut out every crawler.

GuideBy mttrPublished October 16, 2026Updated October 27, 20264 minute read

4.6%Of sites shut out at least one AI crawler by nameOver the 316,711 sites whose robots.txt was asked, as of October 27, 2026
0.2%Of sites shut out every crawler, AI or not713 sites
13,599Sites shut out GPTBot by name, more than any other4.3% of the sites asked
16%Of sites have no robots.txt to read, which blocks nothingOver the 316,711 sites asked

Counted from the 316,711 sites mttr has read, each at its latest read, as of October 27, 2026.

Which AI crawlers are there?

They come in three kinds, and the difference matters more than the names. Some collect pages to train models. Some index pages so an assistant can search them. Some fetch one page because a person asked about it just now. Blocking the first keeps your words out of training; blocking the second keeps you out of the answers.

Token in robots.txt	Run by	What it is for	Obeys robots.txt
`GPTBot`	OpenAI	Collects pages that may train OpenAI's models	Yes
`ClaudeBot`	Anthropic	Collects pages that may train Anthropic's models	Yes
`Google-Extended`	Google	Not a crawler: a token that says whether pages Google fetched may train and ground Gemini	Yes, as a token
`PerplexityBot`	Perplexity	Indexes pages for Perplexity's answers; not for training, Perplexity says	Yes
`ChatGPT-User`	OpenAI	Fetches a page when a person asks ChatGPT to	OpenAI says its rules may not apply
`Claude-SearchBot`	Anthropic	Indexes pages for search in Claude	Yes
`OAI-SearchBot`	OpenAI	Indexes pages for search in ChatGPT	Yes
`bingbot`	Microsoft	Bing's search crawler, whose index Copilot answers from	Yes

Each operator says so on its own page, linked under Sources. Perplexity also runs a user fetcher, Perplexity-User, which it says generally ignores robots.txt, and Anthropic runs Claude-User for the same job. A rule aimed at a user fetcher is a request, not a lock.

How many sites block each AI crawler?

Few, and far fewer than the noise about it suggests. 4.6% of the 316,711 sites mttr has read name at least one AI crawler and shut it out; 0.2% shut out every crawler, AI or not. GPTBot is shut out by name on 13,599 sites and OAI-SearchBot on 2,998: most sites that block are opting out of training, and many are still letting themselves be found.

Sites whose robots.txt shuts out each crawler by name

Crawler	Sites blocking it by name	Sites	With sites closed to all
GPTBotOpenAI	4.3%	13,599	4.5%
ClaudeBotAnthropic	3.6%	11,444	3.8%
Google-ExtendedGoogle	3.1%	9,661	3.3%
PerplexityBotPerplexity	1.9%	5,964	2.1%
ChatGPT-UserOpenAI	1.5%	4,880	1.8%
Claude-SearchBotAnthropic	1.0%	3,297	1.3%
OAI-SearchBotOpenAI	0.9%	2,998	1.2%
bingbotMicrosoft	0.1%	452	0.4%

Counted from the 316,711 sites mttr has read, each at its latest read, as of October 27, 2026. A crawler counts as blocked when the robots.txt rules that apply to it disallow the home page; a rule that closes only part of a site is not counted. The last column adds the 713 sites that shut out every crawler. 16% of these sites have no robots.txt to read, which blocks nothing.

What does robots.txt control, and what does it not?

robots.txt asks crawlers not to fetch paths, one name at a time. The standard, RFC 9309, says plainly that its rules are not a form of access authorization. It is honoured by crawlers that choose to honour it, and the large ones say they do.

It does control whether a well behaved crawler fetches a page, for each token by name.
It does not take back pages already collected, or a model already trained.
It does not bind a fetch a person asked for. OpenAI and Perplexity both say their user fetchers may not follow it.
It does not stop a firewall or a CDN from refusing a crawler, which robots.txt never shows. A site can allow GPTBot in robots.txt and still answer it with a 403.
Google-Extended does not take a site out of Google Search, and Google says the AI features in Search follow the controls Search already has, such as nosnippet and noindex.

Should you block AI crawlers?

Decide training and search separately. If you want an assistant to cite you, let the search and user crawlers in: blocking GPTBot does not take you out of search in ChatGPT, which OpenAI runs on OAI-SearchBot. If you want your words out of training, block the training crawlers by name and say nothing about the rest.

Choose who gets in. The file and what it means follow.

Collect pages to train modelsGPTBotOpenAI. Shut out by name on 4.3% of sitesShut outClaudeBotAnthropic. Shut out by name on 3.6% of sitesShut outGoogle-ExtendedGoogle. Shut out by name on 3.1% of sitesShut out

Index pages for assistants' searchOAI-SearchBotOpenAI. Shut out by name on 0.9% of sitesLet inClaude-SearchBotAnthropic. Shut out by name on 1.0% of sitesLet inPerplexityBotPerplexity. Shut out by name on 1.9% of sitesLet inbingbotMicrosoft. Shut out by name on 0.1% of sitesLet in

Fetch a page a person asked aboutChatGPT-UserOpenAI. Shut out by name on 1.5% of sitesLet in

robots.txt

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

TrainingShut outEvery training crawler named here is asked to stay out.
Assistants' searchFoundAssistants' search can read your pages, so an answer can cite them.
Fetches a person asks forLet inA page a person asks about can be fetched for them.

Shutting out Google-Extended does not take a site out of Google Search.

Each share is of the sites whose robots.txt shuts that crawler out by name. Counted from the 316,711 sites mttr has read, each at its latest read, as of October 27, 2026.

On Cloudflare, a setting called Block AI bots writes rules like these for you, for every AI crawler at once. Check what it wrote before you leave it on.

How do you check your own robots.txt?

The free AI crawler checker reads a site's robots.txt and says which of these crawlers it lets in, and whether there is an llms.txt. A full read with mttr holds the site to the rule Let AI crawlers in among the rest.

See who your robots.txt lets in

Type your domain. The checker reads its robots.txt as each of these crawlers would and says who is let in, and whether there is an llms.txt.

Free. Opens the AI crawler checker with what it finds.

Write the file with your agent

Use mttr to read yoursite.com and tell me which AI crawlers its robots.txt shuts out. Then write me a robots.txt that keeps GPTBot, ClaudeBot and Google-Extended out of training and lets the search and user crawlers in, keeping every rule the file has today for other crawlers. Show me the file before you change it.

Hand this to Claude Code, Codex, Cursor or any agent connected to mttr.

Settle your robots.txt

Decide training and search separately.
Name each training crawler you want out, and only those.
Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot in if you want to be cited.
Keep a Sitemap line with the full address of your sitemap.
Check that your firewall or CDN does not refuse the crawlers robots.txt lets in.
Read the file again after any setting that writes it for you.

Sources

Overview of OpenAI crawlersGPTBot, OAI-SearchBot and ChatGPT-Userdevelopers.openai.com
Does Anthropic crawl data from the web?ClaudeBot, Claude-SearchBot and Claude-Usersupport.claude.com
Perplexity crawlersPerplexityBot and Perplexity-Userdocs.perplexity.ai
Google's common crawlersGoogle-Extended and what it does not affectdevelopers.google.com
AI features and your websiteGoogle Search Centraldevelopers.google.com
RFC 9309: Robots Exclusion ProtocolThe standard for robots.txt, 2022rfc-editor.org

How the figures were counted. Every figure on this page is counted from the 316,711 sites mttr has read, each at its latest read, with drafts left out, as of October 27, 2026, and worked out again as sites are read. A share of pages counts only pages that were read: a site read only in part speaks for the pages read and no others. Each rule is the one in mttr's rulebook, decided by code. A score or a rule passed says pages are well formed. It does not say an assistant cites the site. The methodology explains each count.