Everyone has an opinion on whether websites are blocking AI crawlers. We wanted the numbers, so on 4 November 2026 we downloaded the Tranco ranking of popular domains, took the top 2,000, fetched each site's robots.txt, and parsed the 1,069 that gave us a usable file for rules about 13 AI crawlers. We also checked whether each site serves an llms.txt. The per-site results are in a CSV you can download .
The short version
About 1 in 5 sites blocks the big training crawlers: GPTBot (19.7%), ClaudeBot (18.8%) and CCBot (20.7%). 25.1% block at least one of six major training crawlers.
Far fewer block the crawlers that decide whether you can appear in an AI answer: OAI-SearchBot (6.7%), Claude-SearchBot (8.1%) and PerplexityBot (11.6%). 12.3% block at least one of the three.
Blocking GPTBot is not blocking ChatGPT search. Of the 211 sites that block GPTBot, 139 (65.9%) still allow OAI-SearchBot.
Blocking is flat across rank.…