AI crawler checker
Which AI crawlers a site’s robots.txt lets in, from GPTBot and ClaudeBot to PerplexityBot, and whether it has an llms.txt.
What does it check?
Rules from the rulebook, and how many of the sites mttr has read miss each.
- Serve a robots.txtServe a plain text /robots.txt.16%of sites miss it
- Let AI crawlers inRemove the robots.txt rules that shut out AI crawlers. On Cloudflare, also turn off "Block AI bots", which writes these rules for you.5%of sites miss it
- Stop refusing AI crawlersStop the firewall or CDN from refusing AI crawlers. This test only imitates their names, so confirm in the CDN dashboard.
- Add an llms.txtAdd /llms.txt: a short plain text map of the site for assistants. Its effect is unproven, so treat it as cheap insurance.80%of sites miss it
example.com/robots.txt
# Search and answer engines that cite their sources
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /
# Crawlers that read for training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlEach AI crawler is named on its own line, and nothing under its name or under * shuts it out of the pages meant to be found.
Questions
Which AI crawlers are there?
GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, and more. Each is named in robots.txt on its own.
Does blocking GPTBot keep my site out of ChatGPT?
It keeps OpenAI from reading pages to train on. ChatGPT’s search reads under OAI-SearchBot, a separate name with its own line in robots.txt.
Does anything read llms.txt?
Some agents and tools do, when asked to read a site. No major search engine has said it uses one.
The whole site
One page is one part. A read checks every page against the whole rulebook and says what to fix first.
Read a whole site