LeaderboardEvery site, rankedFree toolsChecks, no sign-upAgentsYour agent fixes itDocsThe API and MCP

Guides3 min readUpdated

Let crawlers in

When a site keeps crawlers out, nothing else in a report matters: search engines and assistants cannot read it, so the score is held at 20 or less. Here is how to open it, host by host.

This page as Markdown

Why can’t search engines read my site?

Something tells crawlers to stay out, often left over from while the site was being built: a robots.txt that disallows every crawler, a noindex on the pages, or a setting at the host such as a bot challenge, a password, or a toggle that blocks AI crawlers. Until it is open, the score is held at 20 or less.

What shuts a site

What shuts itWhere it is setWhat to look for
robots.txtA file at the site’s root, or a setting that writes itDisallow: / under User-agent: *. Many builders write this while a site is being made and leave it there
A noindexA tag on the pages, or a header from the server<meta name="robots" content="noindex">, or X-Robots-Tag: noindex
A setting at the hostThe host’s or the CDN’s dashboardA bot challenge, a password, or a toggle that blocks AI crawlers, which turn crawlers away before they reach a page

A robots.txt that lets everyone in, and names the sitemap, looks like this:

robots.txt
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Then read it again

Press Read it again on the report once the change is live. A cached robots.txt can take a few minutes to clear.

Steps for your host

Cloudflare

  1. Under Security, then Bots, turn off Bot Fight Mode. It answers crawlers with a challenge page instead of the site.
  2. In AI Crawl Control (or Security, then Bots, on older dashboards), turn off "Block AI bots" if you want assistants to read the site.
  3. Turn off the managed robots.txt if Cloudflare is adding rules to yours. Your own file is then served as it is.
  4. Under Security, then WAF, look for custom rules that block by user agent or country, and make sure "I’m Under Attack" mode is off.

WordPress

  1. Settings, then Reading: untick "Discourage search engines from indexing this site". It adds both a noindex and a Disallow: /.
  2. If an SEO plugin edits robots.txt (Yoast, Rank Math, All in One SEO), check its robots.txt editor for Disallow: /.
  3. A real robots.txt file in the site’s root folder overrides both. Open it in your host’s file manager and fix it there.

Namecheap

  1. Shared hosting: open cPanel, then File Manager, then public_html, and edit robots.txt. Delete the file if you are unsure; no robots.txt lets every crawler in.
  2. EasyWP: the site is WordPress, so follow the WordPress steps from its dashboard.
  3. If only the domain is at Namecheap and the site is hosted elsewhere, the fix is at the host. Pick that host instead.

GoDaddy

  1. Websites + Marketing: in the site’s settings, check that it is published and that search engine visibility is on.
  2. cPanel hosting: File Manager, then public_html, then robots.txt, as with Namecheap.

Squarespace

  1. Settings, then Site visibility: the site must be Public, not password protected.
  2. Settings, then Crawlers: turn off blocking of AI crawlers if you want assistants to read the site.
  3. For each page, under its SEO settings, make sure "Hide page from search engine results" is off.

Wix

  1. Dashboard, then Settings, then SEO: turn on "Let search engines index your site".
  2. In the same place, open the robots.txt editor and remove Disallow: /.

Shopify

  1. Online Store, then Preferences: remove the storefront password. A store behind its password page cannot be read.
  2. If the theme has a robots.txt.liquid template, check it for rules that disallow the whole store.

Webflow

  1. Site settings, then SEO: check the robots.txt field for Disallow: /, and that the site is published to its own domain.
  2. Each page’s settings: turn off "Exclude from search".

Vercel, Netlify and your own server

  1. robots.txt is a file in the site’s public folder. Edit it there and deploy.
  2. Check the server’s headers for X-Robots-Tag: noindex, in vercel.json, netlify.toml, _headers, or an nginx add_header line.
  3. Protection on a deployment (a password or a sign-in wall) keeps crawlers out of that address. The address the public uses must be open.

Sources