Menu

1 / 5

How many small business websites block AI crawlers?

DEV Community: datascience·Preferly Team·about 2 hours ago
#x8QqoODo
#devtodatascience#science#devto#dev#sites
Reading 0:00
15s threshold

Short answer: about 1 in 7 small-business websites aren't fully open to AI crawlers, and most owners didn't choose that. Only about 3% block AI crawlers on purpose in robots.txt.

This is a cross-post of Preferly's October 2026 study . The method, charts and category totals live there, and the aggregate data is free to reuse under CC BY 4.0 .

We checked 4,111 websites of small and local businesses in 26 US cities, and could analyze 3,693 of them. 85.6% were fully open to AI crawlers. The other 14.4% weren't: 5.8% showed our clearly identified crawler a bot check, 5.0% didn't load at all (mostly because of broken HTTPS certificates), 3.1% block some or all AI crawlers in robots.txt, and 0.4% had a robots.txt we couldn't read. Only the robots.txt blocks are a rule written for crawlers. The bot checks come from hosting and security vendors, and the broken homepages are certificate and server errors, so most of those owners never decided anything about AI.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.