Menu

Your robots.txt probably does not say what you think it says about AI crawlers
πŸ“°

Your robots.txt probably does not say what you think it says about AI crawlers

DEV Community: webdevΒ·orenhacham-labΒ·1 day ago
#wR0nFcE5
#devtowebdev#devto#dev#group#disallow
Reading 0:00
15s threshold

I read 700 of the highest-ranked sites' robots.txt files on 9 November 2026, matched each one

against the AI crawler tokens their operators document, and published the whole reading β€” domain

by domain, with the exact line that decided each verdict β€” as open data. 407 of the 700 files

could actually be read; 145 hosts refused the request outright. Those are large, high-ranked

sites, not small businesses, so do not read the shares below as a picture of small business.

Two numbers from that reading are the reason for this post.

Of the sites that block GPTBot, 80.3% wrote a rule that names it. That block was a decision:

somebody sat down and kept their pages out of a training corpus.

Of the sites that block OAI-SearchBot β€” the crawler that fetches a page at the moment someone

asks a question, in order to answer with it and link back β€” 43.6% never chose it at all.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.