Menu

1 / 2

Is your site invisible to ChatGPT? robots.txt, llms.txt and JavaScript checks for AI crawlers

DEV Community: webdev·Ed Grows·1 day ago
#pZcBeoJu
#devtowebdev#devto#dev#robots#search
Reading 0:00
15s threshold

If ChatGPT, Claude or Perplexity never mention your product, the reason is often boring and fixable: the assistant cannot read your homepage in the first place. Before you think about content strategy for AI search, check three things on the technical side. Can AI crawlers fetch the page? Does the page contain real text without JavaScript? Does it say clearly what the product is?

I run website.show , a small directory of software built by indie makers, and every listing there gets an automated check of exactly these signals. After reading a lot of these reports, the same handful of mistakes keeps showing up. Here they are, with the fix for each.

Which AI crawlers should you allow in robots.txt?

Short answer: allow the search bots if you want to be quoted in AI answers.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.