Menu

llms.txt on a 4-language static site: how we generate it, validate it and keep it honest
📰

llms.txt on a 4-language static site: how we generate it, validate it and keep it honest

DEV Community: astro·Veilvex·18 days ago
#oTSnrTLg
#devtoastro#devto#dev#llms#price
Reading 0:00
15s threshold

AI assistants now answer "who does X in Y" questions directly, and they build those answers from what they can read about you. For a small team, the cheapest thing you control is your own site: what crawlers are allowed to read, and how clearly your facts are stated.

On veilvex.com (an Astro static site in English, Russian, Azerbaijani and German) we treat this as part of the build, not a one-off file. Here is the setup and the rules we ended up with.

1. robots.txt: say yes explicitly

Many sites block AI crawlers by accident: a template, a CDN toggle, a security plugin. We generate robots.txt at build time and list the crawlers by name, even though User-agent: * already allows everything:

User-agent: * Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: GPTBot Allow: /

User-agent: OAI-SearchBot Allow: /

User-agent: ClaudeBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: Google-Extended Allow: /

Enter fullscreen mode

Exit fullscreen mode

Why list them?…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.