Google, OpenAI and Anthropic adopt unified AI scraping signal

A single technical instruction will soon allow website owners to halt data collection for artificial intelligence models across four of the industry's largest players. Google, Microsoft, OpenAI and Anthropic have agreed to implement a new user-agent rule within the robots.txt file, enabling publishers to opt out of content scraping specifically for generative AI training purposes.

The mechanics of the new directive

The proposed standard, currently under review by the Internet Engineering Task Force (IETF), was authored by Kevin Kelley, who serves as Anthropic’s technical lead for standards. The core objective is to simplify the current fragmented landscape where site administrators must identify and configure separate blocking rules for each individual crawler provider. By adopting this common signal, the participating companies aim to recognize one unified instruction that applies across their respective systems.

Under the new protocol, publishers insert a specific section into their robots.txt file designated for AI training agents. The syntax requires defining a user agent labeled "__usage_ai-training" followed by disallow or allow paths. For example, a publisher might use "Disallow: /" to block all pages from being used for model training, while simultaneously using "Allow: /press-kit/" to permit access to specific public relations materials. This structure ensures that general web crawling for search indexing remains unaffected; the directive targets only the specialized bots gathering data for large language models.

Implementation timelines and scope

Each company has outlined distinct schedules for rolling out support for the new standard. Anthropic has already begun honoring the directive. Microsoft stated its systems will start recognizing the signal on February 1, 2027. Both Google and OpenAI indicated they will implement the changes in early 2027, with OpenAI specifying a timeline of "the coming months."

It is important to note the limitations of this control mechanism. The directive operates prospectively and cannot reverse or delete content that has already been scraped and stored by these companies. If a page was crawled prior to the implementation of the new rules, the data remains in the companies' possession. The change strictly prevents future automated collection of that specific content for training purposes. Google clarified that it interprets the new user-agent tag as a stop sign specifically for its "Google-Extended" crawler, which is responsible for gathering materials used to train Gemini models.