Menu

More News Sites Default To Blocking AI Crawlers
📰
0

More News Sites Default To Blocking AI Crawlers

Search Engine Journal·Matt G. Southern·3 months ago
#XytRfwc2
Reading 0:00
15s threshold

Reuters and Time now default to blocking AI bots, allowing only approved crawlers through allowlists, Digiday reports . Both publishers made the decision in May, joining People Inc. and The Atlantic, which adopted similar setups within the past year. Reuters says the change hasn’t cost it traffic, while cutting what it spends serving bots. Executives credit the added friction with helping push AI companies toward licensing talks. Why Blocklists Weren’t Enough Robots.txt works only when crawlers choose to honor it. Digiday cited a Tollbit report finding that 30% of total AI bot scrapes didn’t comply with explicit robots.txt permissions. Blocking at other levels still has teeth, the executives say . Scrapers that route around blocks pay for workarounds, and that expense is the point. A blocklist catches only the bots a publisher can name. People Inc. learned that switching to an allowlist increased the number of user agents it blocked from about 2,100 to more than 30,000.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More