# How mttr reads a site

Updated October 24, 2026.

What it fetches, in what order, what it keeps, and what it will not do on anyone’s site.

## What does mttr fetch from a site?

Only its public pages, by their public address. It checks the domain, reads `robots.txt` and obeys it, reads the sitemap, then reads the home page and follows links, a few pages at a time, up to 100 pages. Every request names itself as mttr, and nothing behind a sign-in is read.

## One read, start to finish

1. **The address.** mttr checks the domain resolves to a public address. A private address is never read.
2. **robots.txt.** It reads `robots.txt` and follows it, for its own name and for every crawler. A site that keeps all crawlers out gets one page read and a report that says why.
3. **The sitemap.** It reads the sitemap the site names, so pages no link leads to are still found.
4. **The pages.** It reads the home page, then follows links and the sitemap, a few pages at a time, up to 100 pages.
5. **The rules.** Every page is checked against one rulebook. Code decides each rule. Where a model is asked for an opinion and is unsure, the page goes to a person instead of passing or failing.

## How it names itself

Every request carries the user agent `mttr/0.1 (+https://mttr.si)`, so a server’s logs show each visit and where to learn what it was. In `robots.txt` it keeps to the rules written for `mttr` when there are any, and to the rules for every crawler when there are none. A path those rules keep it from is never fetched and never counted in the report.

## What is kept

| What | How often | Why |
| --- | --- | --- |
| The summary | Every read | The score, the four stages, and each rule with how many pages fail it |
| Every page | Every read | Its status, depth, words, links, and the rules it fails |
| Its text | Once a version | A page whose words have not changed adds nothing new |
| Its HTML | The newest only | So a change to the rulebook can be scored again without fetching the site |

## What it will not do

- Read a private address, a page behind a sign-in, or a port other than 80 or 443.
- Pretend to be anyone else. Every request names itself as mttr.
- Go where `robots.txt` says it may not.
- Read one site more than once every 10 minutes.
- Report pages no link leads to when the read stopped at its limit, since it cannot tell those from pages it did not reach.

## Keeping mttr out

> **To keep mttr out.** Add `User-agent: mttr` and `Disallow: /` to your `robots.txt`. It is obeyed on the next read.

## Sources

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), IETF: How robots.txt names a crawler and what it may read.
- [Sitemaps XML format](https://www.sitemaps.org/protocol.html), sitemaps.org: The sitemap format a read takes its list of pages from.
- [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), OpenAI: The names of OpenAI’s crawlers.
- [Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518), Anthropic: The names of Anthropic’s crawlers.
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots), Perplexity: The names of Perplexity’s crawlers.