How mttr reads a site

What does mttr fetch from a site?

Only its public pages, by their public address. It checks the domain, reads robots.txt and obeys it, reads the sitemap, then reads the home page and follows links, a few pages at a time, up to 100 pages. Every request names itself as mttr, and nothing behind a sign-in is read.

One read, start to finish

1. The address

   mttr checks the domain resolves to a public address. A private address is never read.

2. robots.txt

   It reads robots.txt and follows it, for its own name and for every crawler. A site that keeps all crawlers out gets one page read and a report that says why.

3. The sitemap

   It reads the sitemap the site names, so pages no link leads to are still found.

4. The pages

   It reads the home page, then follows links and the sitemap, a few pages at a time, up to 100 pages.

5. The rules

   Every page is checked against one rulebook. Code decides each rule. Where a model is asked for an opinion and is unsure, the page goes to a person instead of passing or failing.

How it names itself

Every request carries the user agent mttr/0.1 (+https://mttr.si), so a server’s logs show each visit and where to learn what it was. In robots.txt it keeps to the rules written for mttr when there are any, and to the rules for every crawler when there are none. A path those rules keep it from is never fetched and never counted in the report.

What is kept

What it will not do

• Read a private address, a page behind a sign-in, or a port other than 80 or 443.
• Pretend to be anyone else. Every request names itself as mttr.
• Go where robots.txt says it may not.
• Read one site more than once every 10 minutes.
• Report pages no link leads to when the read stopped at its limit, since it cannot tell those from pages it did not reach.

Keeping mttr out

Sources