I read 700 of the highest-ranked sites' robots.txt files on 9 November 2026, matched each one
against the AI crawler tokens their operators document, and published the whole reading β domain
by domain, with the exact line that decided each verdict β as open data. 407 of the 700 files
could actually be read; 145 hosts refused the request outright. Those are large, high-ranked
sites, not small businesses, so do not read the shares below as a picture of small business.
Two numbers from that reading are the reason for this post.
Of the sites that block GPTBot, 80.3% wrote a rule that names it. That block was a decision:
somebody sat down and kept their pages out of a training corpus.
Of the sites that block OAI-SearchBot β the crawler that fetches a page at the moment someone
asks a question, in order to answer with it and link back β 43.6% never chose it at all.β¦