Do you know which bots your CDN is blocking? Robots.txt is voluntary, and real enforcement may be happening without your SEO team knowing.

For years, bot management in SEO was relatively straightforward. You either allowed crawlers to access your website or blocked them. That model was never perfect, but it was generally sufficient for what we needed.

Image from author, November 2026

AI search has made that approach increasingly inadequate.

Today, websites are accessed by traditional search crawlers, AI training crawlers, retrieval bots, user-initiated fetches , and autonomous agents.…