# Methodology

Updated October 24, 2026.

How mttr reads a site, scores its pages, ranks it against other sites, places it in a line of work and looks for it in an assistant’s answers. Every figure here comes from the code that does the work.

## How does mttr reach a score?

It reads up to 100 public pages of a site under its own name, checks every page against one rulebook of 44 rules in four stages, and adds the stages by their share of 100 points. Code decides every check. Where a model is asked and is unsure, the page waits for a person. A score says pages are well formed. It does not say an assistant cites the site.

| Step | What happens | Decided by |
| --- | --- | --- |
| Read | Up to 100 public pages, 6 at a time, `robots.txt` obeyed | Code |
| Check | 44 rules, each a check on the site or on every page it applies to | Code, and a model for 2 rules |
| Score | Each stage the weighted share of its checks that pass, the four added by their share | Code |
| Rank | A place among the 261,380 ranked sites in the US and Canada | Code |
| Place | A line of work, from what the site’s pages mean | Code, on a model’s vectors |
| Cite | Whether an assistant’s answer lists the site among its sources | Code, from the sources |

## How is a site read?

Only by its public address, with every request named `mttr/0.1 (+https://mttr.si)`. mttr reads the HTML each page’s server sends, as a crawler does first, and does not run the page’s scripts, so words written in later by script are not seen. The [docs on reads](https://mttr.si/docs/reads) give the same steps in short.

1. **The address.** The domain must resolve to a public address. Every fetch checks it again, redirects too, so a name that points inside a network is never read. Only http and https on ports 80 and 443 are fetched, never an address with a sign-in in it.
2. **robots.txt.** It is fetched before any page and obeyed path by path, as RFC 9309 says: the rules written for `mttr` when there are any, else the rules for every crawler. The longest matching rule decides, Allow wins a tie, and a path no rule matches may be read. A path is asked with and without its last slash, and kept out if either is.
3. **The sitemap and llms.txt.** Every sitemap `robots.txt` names is read, and `/sitemap.xml` as well. `/llms.txt` is fetched to see whether there is one.
4. **The home page.** It is read first and by itself, since the rules treat it as the home page. It is always read, so a report can say when a site is closed.
5. **The pages.** Then the sitemap’s pages in its order, then the pages they link to, 6 at a time. When one comes back the next goes out, so a slow page holds up only itself. The read stops at 100 pages.
6. **The four forms of the address.** While the pages are read, `http`, `https`, `www` and the bare name are each asked for once, to see that all four end at one address.

| Limit | Value |
| --- | --- |
| Pages in a read | 100 |
| Pages of one site fetched at once | 6 |
| Time a page has to answer | 20 seconds |
| Redirects followed | 5 |
| Reads of one site | One every 10 minutes, whoever asks |
| A site closed to every crawler | The home page alone |

## What a read skips

| What | What happens |
| --- | --- |
| A path `robots.txt` keeps mttr from | Never fetched and never counted. A redirect that lands on one is dropped and its place given back |
| Images, PDFs, stylesheets, scripts, data, archives, media, fonts and office files | Never fetched, known by the end of the address |
| An answer that is not HTML, such as Markdown or plain text | Not graded and its links not followed. It still takes a place in the read |
| One page at two addresses, by a last slash or a redirect | Read once, and the second place given back |
| A page that fails to load | Counted once, by the rule that pages load, and by no other rule |
| A page behind a sign-in, a form, or an address inside a network | Never read |

> **Never more than was read.** A read that stops at 100 pages cannot tell a page no link leads to from one it did not reach, so it reports neither orphans nor gaps in what the site covers. The report says it stopped. See [Stopped at the limit](https://mttr.si/docs/limit).

## What does the rulebook check?

44 rules in four stages, in the order a crawler meets a site. 13 are about the site as a whole, 29 are checked on each page they apply to, and 2 are read by a model. Each rule has a weight from 1 to 5 and a fix. [The rulebook](https://mttr.si/rulebook) lists every one, and this page does not restate them.

| Stage | What it asks | Rules | Share of the score |
| --- | --- | --- | --- |
| Access | Can it be fetched and indexed at all? | 16 | 35% |
| Understand | Can a machine tell what each page says, and lift an answer from it? | 17 | 30% |
| Trust | Is there a reason to believe it, and to cite it? | 9 | 20% |
| Act | Can the visitor, or an agent, take the next step? | 2 | 15% |
| All |  | 44 | 100% |

## How a stage scores

A rule about the whole site is one check. A rule about pages is one check on every page it applies to, so a rule on 100 pages is 100 checks. A stage is the share of its checks that pass, each counted by its rule’s weight, rounded to a whole number. A heavy rule failing on many pages costs the most.

- A check that waits for a person is left out until a person reads it, so it neither raises nor lowers a stage.
- A rule that throws on a page holds that page for a person, and the read goes on.
- A stage with no checks counted, such as one whose rules apply to no page read, counts as 100.

## How the stages add up

Each stage is multiplied by its share, the four are added, and the sum is rounded. For a site whose stages came out like this:

| Stage | Stage score | Its share | Adds |
| --- | --- | --- | --- |
| Access | 92 | 35% | 32.2 |
| Understand | 75 | 30% | 22.5 |
| Trust | 60 | 20% | 12.0 |
| Act | 100 | 15% | 15.0 |
| Score |  |  | 82 |

## When a site is closed

3 rules can close a site: let crawlers in, fix pages that fail to load and allow pages to be indexed. One closes it when it fails for the whole site, on the home page, or on more than half the pages read. A closed site’s score is held at 20 or less, whatever else passes, since a score for pages nobody can reach would mislead. One broken link or one page kept out of search on purpose is a fix, not a closed site.

## Rules that apply to some sites

- By the kind of page. Each page is taken as the home page, an article, a service or another page, from its address and its structured data. 13 rules ask only of some kinds, and 5 only of articles, such as showing when a guide was updated. Kinds are guessed, and a service page at an address that does not say so can be asked fewer rules than it should.
- By what the site is. 1 rule asks only of some sites. A tap-to-call link is asked only of a site that serves a place: one that writes out a phone number or a street address, or gives a phone, an address or a local business in its structured data.
- By whether it is live. A draft read sets aside the 3 rules a site not yet open cannot pass, such as a sitemap search engines can see, so its score says how ready it is for the day it opens.

## Rules a model reads

2 rules need reading, not counting, such as whether a page’s first paragraph answers the question above it. A model reads those pages and gives a verdict with how sure it is. Code then decides what the verdict is worth.

| The model’s reply | What mttr does |
| --- | --- |
| In the form asked for, finished, and at least 70% sure | Counts it: the page passes or fails the rule |
| Not in the form asked for, or cut off | Holds the page for a person |
| Less than 70% sure | Holds the page for a person, with what the model said |

> **A model gives opinions, code decides.** A model’s answer never passes or fails a page on its own, and a page it was unsure of counts for nothing until a person reads it. See [Needs a person](https://mttr.si/docs/person).

## How fixes are ordered

Each failing rule is a fix. Fixes are ordered by the rule’s weight times the share of its checks that fail, so a heavy rule failing on a few pages can sit below a light one failing on most. A rule that is closing the site always comes first. Each fix shows the points the score would gain if every page passed it, worked out from the same counts the score is.

## How are sites ranked?

By score, among the latest read of every ranked site. 261,380 sites are ranked today. A site is ranked when its latest read is neither a draft nor closed to crawlers, and its name is in the market mttr ranks: the United States and Canada. A site outside it keeps its report and is on no board.

| A name ending in | Ranked |
| --- | --- |
| `.us` or `.ca` | Yes |
| A name used everywhere, such as `.com`, `.org`, `.net` or `.io` | Yes |
| Another country’s, such as `.uk` or `.de` | No |
| A region elsewhere, such as `.eu` or `.london` | No |

## Ties

No two sites share a place. Sites on the same score are placed by their standing: how far past the score they go, from how fast and lean their pages are, how much they say and how much of it is their own, how few of their pages show signs of generated text, whether they let AI crawlers in, how they are improving, and how long the domain has been held. Each measure is scored on a fixed curve, never against other sites, so a standing moves only when its own site changes. Its weights are ours and are not published. Past the standing, the site that reached the score first goes ahead, then the name.

## Tiers

A tier is where a site’s place falls among every ranked site. First, second and third each have a mark of their own.

| Tier | Takes |
| --- | --- |
| Top 0.1% | A place in the first 0.1% of ranked sites |
| Top 1% | A place in the first 1% of ranked sites |
| Top 5% | A place in the first 5% of ranked sites |
| Top 10% | A place in the first 10% of ranked sites |
| Top 25% | A place in the first 25% of ranked sites |
| Top 50% | A place in the first 50% of ranked sites |
| Beats a share | Below the top half, the share of ranked sites that score less |

## Boards

The [leaderboard](https://mttr.si/leaderboard) ranks every site, and has a board for each line of work and each state once 20 or more of its sites are ranked. A site’s place on a smaller board is counted the same way, among that board’s sites only. A place in the past counts only the sites read by then.

## How does mttr tell what a site is about?

From its own words. Each page’s main text is cut into passages of about 120 words, each ending at a sentence, and an open model running on our own machines turns each passage into a vector of its meaning. A page gives at most its first 24 passages, about 2,900 words, which is the whole of nearly any page. Text past that is not searched by meaning.

- A page’s meaning is the centre of its passages, and a site’s meaning is the centre of its pages.
- Sites and pages that read alike are close. Close means similar, never better, and never cited.
- A page’s text is not sent to an outside service to be understood.

## Lines of work

There are 16 lines of work, each a short description of the work, from trades and home services to cities and public services. [Why mttr](https://mttr.si/why) shows each one with a question an assistant is asked in it. A site is placed in the one its meaning reads closest to, only when it is at least 0.45 close and 0.015 closer than the next. A site close to none is placed in none. It is an estimate: a site that does two kinds of work is placed by the one it says most about.

## How does mttr tell whether an assistant cites a site?

It asks an assistant the questions people ask, and reads the sources the answer gives. Today that is Gemini, with Google Search on. Only the question is sent: nothing about the site, the person who asked, or mttr. Code, not a model, decides from the sources whether a site was cited.

## The questions

| Part of the bank | What is asked |
| --- | --- |
| Questions in the bank | 2,142 |
| Local lines of work | 9, each asked about 38 cities |
| Cities | 15 large cities across the country, and 23 in the Midwest |
| Lines asked without a place | 7, and search and AI visibility, mttr’s own subject |
| Smaller towns | Trades questions are asked about towns too, the towns claimed sites serve first |
| How often | Each question at most once a week |

Asking costs money every time, so it runs within a daily and a monthly budget, checked before every ask. When the budget is spent, or asking is paused for cost, nothing is asked. Answers are refreshed when they are asked again, and the answers a report shows are the latest kept.

## Deciding a citation

- A site is cited when one of the answer’s sources is its domain or a name under it. Its place is where its first source stands among them.
- A site named in the answer’s words but missing from its sources is a mention, kept apart, and not a citation.
- A report shows only answers to questions about what its site does: a question’s meaning must read 0.30 close to the site’s or closer, and a question that names a kind of business needs the site to be that kind, or 0.50 close when its kind is not known. With nothing that fits, there is no section.

## What the answers show

| Measure | Result |
| --- | --- |
| Sites Gemini answered questions about, set against their scores | 1,890 |
| How often a cited site outscored an uncited one | 85% |

That is a correlation. It does not show that a well formed site is cited because it is well formed, and a site may be cited for being large, known or local, which no rule measures. Each week a model is fit on what cited sites have in common, tested on questions it never saw. Until it orders sites better than the score alone does, nothing it finds moves a weight or a rule.

## A site’s own searches

For a site whose owner connects Google Search Console, mttr asks the searches people really use to find it: its 10 seen most, two words or more, none naming the site. Each is asked at most once a week across sites, with only the question sent. As [the score docs](https://mttr.si/docs/score#cited) say, 7 of the sites run with mttr are cited for their own searches. Results are given as counts, and no such site is named.

> **Well formed is not the same as cited.** A score says pages are well formed. It does not say an assistant cites the site. An assistant answers differently from one day, place and person to the next, so an answer is one sample, not a ranking.

## What mttr does not claim

- That a score brings rank, traffic, citations or sales. It says how well formed the pages read are.
- More than was read. A read that stops at its limit says so, and every figure in it is for the pages it reached.
- That a guess is a fact. A model’s unsure verdict waits for a person, and a line of work is an estimate.
- That one assistant speaks for all. Answers are Gemini’s, on these questions, in these weeks.
- That a figure is real when it is not. A sample on a page is marked as a sample, and a part that is not built says so.
- That a score holds still. The rulebook changes, and History says when two reads were judged by different rulebooks.

## What does mttr keep, and how do I keep it out?

mttr keeps what it reads so a later read can be set against an earlier one, and names sites on its public pages. A site’s owner can have it go by a code, keep mttr out, or have it removed. The [privacy policy](https://mttr.si/privacy) says the rest.

| What | Kept |
| --- | --- |
| The score, the stages, and each rule with how many pages fail it | Every read |
| Each page’s status, depth, words, links and the rules it fails | Every read |
| Each page’s text, and its passages’ vectors | Once a version of the text |
| Each page’s HTML | The newest only, so a changed rulebook can score it again without fetching the site |
| The assistants’ answers and their sources | Each time a question is asked |

- Sites read are named on the leaderboard and in their reports. A site read from its owner’s Search Console, and any whose owner asks, goes by a random eight-letter code instead. The code’s page shows its score, place and stages, never its name.
- To keep mttr out, add `User-agent: mttr` and `Disallow: /` to `robots.txt`. It is obeyed on the next read. See [Keeping mttr out](https://mttr.si/docs/reads#keep-out).
- To have a site removed, or named by a code, ask through [the form on the page for teams](https://mttr.si/enterprise#book). A removed site is not read again.

## Changes to this page

The rulebook is written once, in code, and the figures here are drawn from it, so they change when it does. The day at the top says when these words last changed.

## Read on

- [How mttr reads a site](https://mttr.si/docs/reads.md)
- [What a score means](https://mttr.si/docs/score.md)
- [Needs a person](https://mttr.si/docs/person.md)

## Sources

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), IETF: How robots.txt names a crawler and which paths it may read.
- [Sitemaps XML format](https://www.sitemaps.org/protocol.html), sitemaps.org: The sitemap format a read takes its list of pages from.
- [Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), Google Search Central: Why the HTML a server sends matters before any script runs.
- [Robots meta tag, data-nosnippet and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), Google Search Central: The noindex tag and header that keep a page out of search.
- [In-depth guide to how Google Search works](https://developers.google.com/search/docs/fundamentals/how-search-works), Google Search Central: A page that follows every guideline is still not promised a place in search.
- [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), Google Search Central: Being indexed and eligible is not a promise of being shown in AI answers.