Menu

Server-render the facts: a small-business site AI crawlers can read
📰

Server-render the facts: a small-business site AI crawlers can read

DEV Community: html·Meni Lebovich·6 days ago
#qrMb1bOM
#devto2#devto#dev#user#search
Reading 0:00
15s threshold

Before an AI assistant can name a business, one of its crawlers has to fetch a page and find the facts in it. For most of those crawlers, the page is the HTML your server sends, not what the browser builds afterwards. This is a short, vendor-sourced checklist for making that HTML count.

Know who is fetching

Each vendor runs separate user agents for search, for user-triggered fetches and for model training, and documents them. The search ones are the ones that decide whether a site can appear in answers.

OpenAI: OAI-SearchBot is “used to surface websites in search results in ChatGPT's search features”. GPTBot is for training. ChatGPT-User visits a page when a user asks, and “robots.txt rules may not apply” to it.

Anthropic: Claude-SearchBot indexes for search, Claude-User fetches at a user's request, and ClaudeBot collects training data.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.