AI giants promise publishers a new way to prevent data scraping

OpenAI, Google, Microsoft, and Anthropic have pledged to provide a unified mechanism for publishers to prevent their content from being scraped for AI training, amid growing criticism and legal battles. The new solution will allow a single set of blocking instructions instead of multiple commands per bot, but will not take full effect for months.

On Friday, OpenAI, Google, Microsoft, and Anthropic pledged to provide a unified mechanism for media outlets and other websites to prevent their content from being scraped for training AI models, amid growing criticism and legal battles from news publishers. Currently, publishers must apply multiple blocking commands for each web-crawling bot individually, but the proposed new solution will allow them to enter a single set of instructions, recognized and adopted by any AI company that follows the protocol. However, the solution will not take full effect for months. Microsoft will begin implementing it from February 1, 2027, and Google at some point in 2027. OpenAI, developer of ChatGPT, said it will roll it out "in the coming months," while Anthropic, developer of Claude, stated it has "already begun" rolling it out. This announcement comes as publishers face a surge in bots copying their posts to feed and train AI models. Some entities, such as the New York Times, have turned to the courts. In December 2023, the New York Times filed a lawsuit against OpenAI and Microsoft, accusing them of using millions of articles without permission or compensation to train ChatGPT. Meanwhile, Cloudflare, an internet services provider, has automatically blocked AI-powered software on its new clients' sites since July 2025.