Before an AI assistant can name a business, one of its crawlers has to fetch a page and find the facts in it. For most of those crawlers, the page is the HTML your server sends, not what the browser builds afterwards. This is a short, vendor-sourced checklist for making that HTML count.
Know who is fetching
Each vendor runs separate user agents for search, for user-triggered fetches and for model training, and documents them. The search ones are the ones that decide whether a site can appear in answers.
OpenAI: OAI-SearchBot is “used to surface websites in search results in ChatGPT's search features”. GPTBot is for training. ChatGPT-User visits a page when a user asks, and “robots.txt rules may not apply” to it.
Anthropic: Claude-SearchBot indexes for search, Claude-User fetches at a user's request, and ClaudeBot collects training data.…