The biggest names in AI are backing a new way to opt out of scraping

Publishers would be able to send a single signal expressing their preferences about AI training to many crawlers at once under a proposal from Anthropic that OpenAI, Google, Microsoft, and Anthropic have said they will back. The mechanism is a shared "policy token" put forward by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF).

The token would sit inside a site's robots.txt file, the long-standing location where sites indicate which crawlers may or may not access them for purposes such as search indexing, archiving, or AI training. At present, publishers must communicate their AI training preferences to each AI company individually.

The proposal arrives as publishers contend with a steady flow of AI crawlers gathering their content for use in AI models. Some organizations have turned to other measures: the Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default.

OpenAI, Google, Microsoft, and Anthropic say their AI crawlers will be restricted from accessing a site when they encounter the shared token. Meta communications head Andy Stone indicated support for the setting but did not say whether Meta will honor it. Stone said a common signal that conveys such preferences to multiple providers at once is an important step, and that Meta looks forward to working with other AI companies on the standards and their implementation across the industry.

Microsoft said it will provide granular controls letting publishers opt out of AI training or AI grounding, the process of directing an AI model to a verified source of information to reduce hallucinations. Microsoft noted that sites will still appear in Bing's search results regardless of their preferences. Google has already introduced comparable controls that let sites decide whether their content appears in AI search features or is used for AI training without affecting search visibility.

Adoption timelines differ. Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, and OpenAI said it will implement it in the coming months.

More AI news from TechManNews.