OpenAI, Google, Microsoft and Anthropic announced on 30 October 2026 that they will support a unified robots.txt AI training preference signal. The shared mechanism allows website owners to register an opt-out from automated generative model training across multiple competing developers at once, replacing the requirement to manage individual bot rules for each company.
Key points
- A new cross-vendor token inside robots.txt allows publishers to state AI training preferences across multiple technology providers simultaneously.
- Anthropic engineer Kevin Kelley authored the specification draft, which operates within existing web standards without breaking older crawlers.
- Anthropic has already adopted the signal, OpenAI plans implementation in coming months, and Google and Microsoft will follow next year.
- The system remains voluntary, meaning developers must choose to honour the signal, and private licensing agreements can supersede it.
How the shared preference token works
Website owners have long used robots.txt files to manage search indexers and automated crawlers. The rise of large language models created administrative friction, forcing webmasters to track dozen of crawler identities, such as OpenAI’s GPTBot or Google’s dedicated scraping agents, to prevent automated ingestion of their articles and assets.
The new method resolves that administrative overhead through a specification titled Machine-Readable Training Preference Signal. Authored by Anthropic technical lead Kevin Kelley and submitted to the Internet Engineering Task Force, the draft creates a shared policy token rather than requiring new directives or structural grammar changes.
Site owners apply the token by entering User-Agent: __usage_ai-training into their existing robots.txt file, followed by standard directives like Disallow: /. Because the format relies on the established RFC 9309 standard, legacy web crawlers simply ignore the unknown agent name without failing. General wildcard directives do not trigger the token, preventing accidental blanket blocks of standard search engines.
Kelley wrote the draft to separate content usage preferences from regular traffic management. An instruction forbidding training does not necessarily command a web crawler to halt all retrieval, meaning a site can instruct companies to read pages for live web search while barring the same text from foundational model runs.
The definitions behind model training
The signal relies on definitions set out in A Vocabulary For Expressing AI Usage Preferences, an IETF draft edited by Paul Keller of Open Future and Martin Thomson of Mozilla. That document classifies AI training specifically as using an asset to alter the learned parameters of a generative AI model.
Non-generative tasks remain distinct under this vocabulary. The draft splits automated activity into three distinct areas: AI Training, AI Use, and Search. AI Use refers to supplying content as external context to a model during operation, whereas Search describes standard indexation to help people find source web addresses.
Each category can carry an allowed, disallowed, or unknown setting. If a publisher publishes conflicting directives across different sections of a file, the specification instructs crawlers to respect the most restrictive choice.
The specification documents carry an important caveat: the draft does not technically force compliance. The vocabulary text states plainly that adherence remains up to each recipient, leaving legal enforcement outside the scope of the internet draft itself.
How tech firms plan implementation
The four primary developers behind the agreement plan phased rollouts across late 2026 and 2027. Anthropic confirmed it has already put the shared signal into operation for its data intake systems.
OpenAI announced it will introduce support for the unified rule in the coming months, while maintaining existing crawler settings. In its published crawler documentation, OpenAI maintains distinct identities: GPTBot handles model training, OAI-SearchBot powers search queries inside ChatGPT, and OAI-AdsBot evaluates web pages for commercial advertising.
Google and Microsoft stated they will introduce support during 2027. In a Microsoft post detailing the rollout, the company confirmed publishers will retain the ability to separate training from grounding, ensuring websites can block offline training while still providing real-time data for Bing answers.
Meta stopped short of committing to immediate adoption. Andy Stone, communications director at Meta, said: “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step.” Stone said the company looks forward to working with industry peers on implementation details.
Publishers push for stronger boundaries
The standard arrives amid intense friction between web publishers and artificial intelligence developers over intellectual property. News organisations, book authors and independent platforms have launched dozens of copyright lawsuits against tech firms, alleging mass scraping of public web pages without explicit consent or compensation.
Publishers have struggled to enforce boundaries using defensive infrastructure. While security networks like Cloudflare introduced switches to stop known AI bots by default, new scrapers frequently mask their user identities or ignore robots.txt entirely. Because robots.txt operates as an advisory convention rather than a cryptographic lock, bad actors can bypass the file unless blocked by IP addresses or anti-bot firewalls.
Participants in the IETF drafting group include representatives from News Corp, the BBC, the Financial Times, and the Electronic Frontier Foundation, alongside engineers from Cloudflare and Cisco. The broad industry representation reflects a desire among media companies to establish clearer ground rules before legal disputes reach final court verdicts.
What happens next
The IETF working group on AI preferences will review community feedback on the Kelley draft before its initial expiry date in May 2027. Further revisions will tackle outstanding disputes, including whether providers must pass original content preferences downstream whenever they distribute open model weights to third parties.
For website administrators, existing vendor-specific rules remain valid. OpenAI, Google and Microsoft confirmed that individual crawler entries like GPTBot and Google-Extended will continue to work alongside the new policy token once deployments finish next year.
