Menu

Do German online shops let AI agents in? We checked robots.txt, bot walls and llms.txt on 100 stores
📰

Do German online shops let AI agents in? We checked robots.txt, bot walls and llms.txt on 100 stores

DEV Community·Fast Data·about 8 hours ago
#LSCsNxIF
#dev#shops#agents#robots#client
Reading 0:00
15s threshold

AI shopping agents are coming: ChatGPT, Claude and Perplexity can already browse shops, compare products and soon buy on a user's behalf. But can they actually get into the stores?

We checked 100 large online shops from Germany, Austria and Switzerland (data from 1 November 2026) and looked at three things:

What does robots.txt say about AI crawlers and agents?

Does the homepage even load for a simple automated client?

Is there an llms.txt file that tells AI assistants what the shop is about?

1. robots.txt: most shops let AI bots in

77 of 100 shops have a robots.txt .

Only 9 of them block at least one of the 17 AI user agents we checked (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others).

The strictest: Westwing blocks 15 of 17 , Zalando and Jako-o 8 each, Notebooksbilliger 7.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.