Robots.txt, as the internet standard RFC 9309 defines it, governs which URLs a crawler may request, not what happens to the content afterward, Google’s Gary Illyes said in a Search Central video published Wednesday. To keep one AI crawler out while allowing others, he said, you have to target that crawler’s user-agent token in the file.

The video is the second episode of The Non-Canonical Cabinet , where Illyes and Cherry Prommawin of Google Search Relations take edge-case questions from the community. Illyes is also co-author, with Mozilla’s Martin Thomson, of an IETF Internet-Draft that would give robots.txt a standard way to state usage preferences. Prommawin recommended writing non-English rule paths in percent-encoded form for crawlers other than Google’s.

What Illyes Said About AI Training

The episode’s question was whether robots.txt can allow AI crawling of a website and keep that same information out of generative model training.…