Google releases Google-Extended, allowing website publishers to opt out of data training.

Google has just announced an option for website publishers to choose not to allow Google to use their data to train the company's AI models while still being able to use Google search. This new tool, called Google-Extended, allows websites to continue to be crawled and indexed by web crawlers like Googlebot while avoiding their data being used for training AI models that are constantly evolving.

The company states that Google-Extended will allow publishers to "manage whether their website helps improve Bard and Vertex AI-generated APIs" and adds that website publishers can use a toggle button to "control access to site content". Google confirmed in July that it is training its AI chatbot, Bard, using publicly available data from the internet.

Google-Extended can be configured through the robots.txt file, which informs web crawlers whether they can access specific websites. Google points out that as AI applications expand, they will continue to explore "other machine-readable methods for website publishers to choose and control" and states that more information will be shared soon.

Many websites have already started blocking web crawlers used by OpenAI to scrape data and train ChatGPT, including The New York Times, CNN, Reuters, and Medium. However, there has been controversy over how to block Google. After all, websites cannot completely block Google's web crawlers, or else they will not be indexed in search results. This has led some websites, such as The New York Times, to block Google through legal means by updating their terms of service to prohibit the company from using their content for AI training.