Menu

We checked robots.txt on 1,069 top websites: who blocks AI crawlers, and who still allows ChatGPT search
📰

We checked robots.txt on 1,069 top websites: who blocks AI crawlers, and who still allows ChatGPT search

DEV Community: datascience·Vimal Kumar Jain·6 days ago
#OW5QDaUi
#devtodatascience#science#devto#dev#sites
Reading 0:00
15s threshold

Everyone has an opinion on whether websites are blocking AI crawlers. We wanted the numbers, so on 4 November 2026 we downloaded the Tranco ranking of popular domains, took the top 2,000, fetched each site's robots.txt, and parsed the 1,069 that gave us a usable file for rules about 13 AI crawlers. We also checked whether each site serves an llms.txt. The per-site results are in a CSV you can download .

The short version

About 1 in 5 sites blocks the big training crawlers: GPTBot (19.7%), ClaudeBot (18.8%) and CCBot (20.7%). 25.1% block at least one of six major training crawlers.

Far fewer block the crawlers that decide whether you can appear in an AI answer: OAI-SearchBot (6.7%), Claude-SearchBot (8.1%) and PerplexityBot (11.6%). 12.3% block at least one of the three.

Blocking GPTBot is not blocking ChatGPT search. Of the 211 sites that block GPTBot, 139 (65.9%) still allow OAI-SearchBot.

Blocking is flat across rank.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.