🔍 Read the full analysis: The Shared Opt-Out For AI Training Has Limits. Here’s What They Are on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced plans to recognize a shared robots.txt signal asking them not to use a website’s content for generative AI training. The signal is voluntary, has staggered implementation dates and does not cover AI answers generated from live web content; Meta has voiced support but has not committed to honoring it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will recognize a shared robots.txt signal asking companies not to use a website’s content to train generative AI models. The move could reduce the need for publishers to set separate training preferences for each company, but the signal is voluntary, does not block AI answers based on live web content and will not take effect for every company at the same time.
The proposed rule uses a shared user-agent label, __usage_ai-training, followed by Disallow: / in a site’s robots.txt file. Its purpose is to communicate a publisher’s preference against training use. The technical basis is an Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Kevin Kelley of Anthropic and published on October 30. An Internet-Draft is a proposal, not an adopted internet standard.
The companies have described different rollout plans. Anthropic says it has already started implementing support, though it expects implementation across its services to take time. OpenAI says it aims to support the signal “in the coming months.” Microsoft says it will honor it from February 1, 2027, as it crawls websites. Google says it aims to support the control in 2027; its crawler documentation specifies early 2027.
Google says that if a publisher uses the shared signal and has no separate rule for Google-Extended, it plans to treat the shared signal as a disallow for Google-Extended. Microsoft is also introducing a separate page-level preference for “AI Grounding,” covering web content used to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences begin February 1, 2027. Its existing NoArchive and NoCache tags will stop counting as training or grounding preferences after March 1, 2027.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A Simpler Signal, Not a Block
For publishers, a shared rule could make it easier to state a training preference without maintaining a separate instruction for every participating company. That matters as AI providers use web content in different ways and sites face a growing set of crawler controls. The change offers a common signal, but it does not give publishers a technical guarantee that every company will comply.
The distinction between training and answers is important. The proposed signal addresses use of content to train generative AI models; it does not, by itself, prevent a service from crawling a site or using current web pages to inform an AI-generated answer. Microsoft’s separate grounding preference highlights that these are distinct choices. Publishers may need to manage training, AI answers and search visibility separately, while keeping existing company-specific controls in place during the rollout.
The signal also creates no payment right and does not guarantee traffic or inclusion in search results. Microsoft says neither choice guarantees “inclusion, traffic, or payment.” The announcement is a coordination step, not a new compensation system or a promise that opting out will remove material from models that have already been trained.
Products Worth Considering
DJI Osmo Mobile 7P Phone Gimbal with Mic Mini Transmitter, Infinity Black
Bose QuietComfort Wireless Noise Cancelling Headphones, Sandstone
Jackery Explorer 300 Portable Power Station, 292Wh
As an affiliate, we earn on qualifying purchases.
From Separate Crawler Rules to One
Before this proposal, a site seeking to express training preferences generally had to account for company-specific controls. Examples include GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft has used page tags such as NoArchive and NoCache for related preferences. The new signal is intended to provide a shared instruction, while companies’ announcements say existing controls will continue to apply during the change.
The proposal is connected to the Internet Engineering Task Force’s AI Preferences working group, known as AIPREF. But the document remains an individual Internet-Draft and has no formal standing in the IETF standards process. At a working-group meeting on October 21, one of the chairs said, “We haven’t declared consensus on anything yet.” Company support for the proposal therefore should not be confused with formal standards approval.
Meta’s communications chief, Andy Stone, said the company supports publishers’ ability to express preferences and called a common signal an important step. That statement did not commit Meta to honoring the signal. The source material also reports no commitment from Apple, Amazon, xAI or Perplexity, so the proposal’s reach beyond the four announcing companies is not established.
““has already started implementing support for this draft””
— Anthropic, in its announcement about the draft
Coverage and Compliance Remain Open
The signal is a request, not a technical lock. The announcements do not establish how compliance will be monitored or what recourse a publisher would have if a company ignored the preference. It is also unclear how many additional AI providers will adopt the rule, or whether Meta will move beyond its statement of support.
Implementation timing remains uneven: Anthropic says rollout has begun, OpenAI gives a target of the coming months, and Google and Microsoft point to 2027 dates. The precise coverage across each company’s services is not fully detailed in the announcements. The proposal’s standards status may also change as the IETF working group considers it; it had not declared consensus at the meeting cited in the source material.
Watch Rollouts and Standards Review
The next developments are the companies’ actual deployments and updates to their crawler documentation. Publishers will need to check each provider’s instructions rather than assume that one shared line is already recognized everywhere. Microsoft’s stated start date is February 1, 2027; its notice says NoArchive and NoCache cease to count as training or grounding preferences after March 1, 2027.
The IETF working group may continue reviewing the draft, but the source material does not give a timetable for formal adoption. Further statements from Meta and other AI providers could clarify how broadly the signal will be honored. Until then, the announced plan is a common preference for training use—not a universal opt-out from crawling, AI-generated answers or use of content already incorporated into trained models.
Key Questions
What does the shared AI training signal do?
It is a proposed robots.txt instruction that communicates a publisher’s preference not to have its website content used to train generative AI models. It does not technically prevent access to the site.
Does the signal stop AI companies from using my site in answers?
Not on its own. The proposed rule covers training use, not AI answers based on live web content. Microsoft has announced a separate AI Grounding preference for that type of use.
When will the companies honor it?
Anthropic says implementation has started, OpenAI aims for the coming months, Google aims for 2027, and Microsoft’s stated start date is February 1, 2027. The schedules and service coverage differ.
Has Meta committed to honoring the signal?
No commitment is reported in the source material. Meta communications chief Andy Stone expressed support for publishers’ ability to state preferences, but did not say whether the company would honor this specific signal.
Is this an official internet standard?
No. It is an Internet-Draft associated with the IETF’s AI Preferences working group. The draft has no formal standing in the IETF standards process, and the working group had not declared consensus at the October 21 meeting cited in the source.
Source: ThorstenMeyerAI.com
