Google announced plans for a major new feature, one that will give publishers and website owners far more control over how AI interacts with their content.
AI has wreaked havoc on many publishers and websites, as many people use the technology in place of traditional search engines. Unfortunately for publishers, AI often still uses their content to generate answers, but without necessarily driving traffic to the website providing that content. There have been efforts to improve the situation, such as incorporating source links in the AI-generated answers, but those measures have had only limited success.
In response, Google is now joining Anthropic, Bing, and OpenAI in unveiling additional controls that websites and publishers can use to better manage their content and how AI utilizes it.
The company described how the new controls work in a developer blog post :
Instead of managing separate instructions for every AI developer, website owners will be able to use a shared ‘policy token’ within robots.txt…