Originally published on We0 on November 16, 2026. This English edition was adapted with AI assistance.
Crawler permissions are becoming an operational concern for technical SEO. Inventory actual traffic, distinguish search indexing from training and grounding, and validate each provider’s controls before changing a sitewide rule.
Keep Search Access Separate from Other Uses
Start from your content policy and verify the tokens your target providers currently support. The example below permits Google Search crawling while declining uses governed by Google-Extended; it is not a sitewide access-control mechanism.
User-agent: Googlebot Allow: /
# Training and grounding uses: declare according to policy User-agent: Google-Extended Disallow: /
Enter fullscreen mode
Exit fullscreen mode
Pay particular attention: a Disallow for Google-Extended will not make a page disappear from Google Search results.…