Shared robots.txt Token Proposed for AI Training Opt-Outs

Introduction

OpenAI, Google and Microsoft have announced support for a shared way for websites to say "do not use my content for AI training" in robots.txt (Google's and Microsoft's posts are dated October 30, 2026). The mechanism comes from an Internet-Draft, draft-kelley-aipref-training-preference-signal-00, authored by K. Kelley of Anthropic and dated 30 October 2026. It binds the preference to a per-purpose "policy token" placed in an ordinary robots.txt User-Agent line, and it "introduces no new directive names and no new grammar."

This is an Internet-Draft, not a ratified standard. The IETF's own boilerplate says such documents are working documents and that it is inappropriate to cite them other than as "work in progress." The draft lists its intended status as Standards Track and expires on 3 May 2027 unless updated.

How the Signal Works

The draft defines one token, __usage_ai-training. Its introductory example reads:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft explains that this expresses a preference against AI training for all paths except those under /press-kit/, where it expresses a preference in favor. Key rules from the text:

• Existing syntax only. Path language, longest-match precedence and group structure from RFC 9309 are unchanged. Parsers that do not implement the mechanism treat the group as they would any group for an unrecognized user agent.
• Wildcards do not match. A User-Agent: * group "MUST NOT be interpreted as matching the policy token." If no group names __usage_ai-training, no preference has been expressed.
• Exact matching. Consumers must match the token exactly, apart from case, and one group may name several policy tokens.
• Separate from crawling rules. A crawler still checks the rules for its own product token exactly as before. The draft states the token "is not a directive for crawling behavior," though a consumer may refrain from fetching content it could not use.
• Precedence. Specific expressions should take precedence over general ones, consumers may treat their proprietary policy tokens as more specific, and the draft says "Private agreements MAY override" preferences expressed this way.
• Definition. "AI Training" follows the IETF AI Preferences working group vocabulary draft: using an asset to modify the learned parameters of a generative AI model. Models used exclusively for non-generative tasks are excluded.

What Each Company Says

Details differ. Google says that if a site opts out of training with the token and does not explicitly specify Google-Extended, it plans to interpret that as if the publisher had disallowed Google-Extended. OpenAI says the new signal does not replace the controls it already offers. Microsoft pairs the shared training preference with a separate page-level grounding preference with Allow, DisallowExtended and Disallow options, and says these settings do not change eligibility for Bing search results. Microsoft also says a grounding opt-out will reduce brand visibility in AI experiences and that the preferences do not establish licensing or payment arrangements.

The Verge reports that Google and Microsoft plan to support the signal "next year" and OpenAI in the "coming months." Those timelines match the vendors' own posts above; where a vendor post is more specific, this article follows the vendor.

Practical Implications for Site Owners

Under the draft, publishing the group above would express a site-wide opt-out for participating consumers once they implement it. A few practical points follow from the sources:

1. Implementation is still ahead. Per their own posts, OpenAI's implementation is coming in the next months, Google aims for 2027, and Microsoft begins honoring preferences on February 1, 2027.
2. Existing controls mostly remain. Google and OpenAI say their current mechanisms continue alongside the new signal. Microsoft is different: it says the new preferences replace NoArchive and NoCache for training and grounding.
3. Absence is not consent or refusal. A robots.txt without the token expresses no preference under the draft, and User-Agent: * does not count.
4. Microsoft publishers have a deadline. Sites relying on NoArchive or NoCache for training or grounding are asked to publish new preferences by March 1, 2027.
5. It governs use, not access. Microsoft notes that an Allow setting does not override a robots.txt rule blocking its crawler.

Pros and Cons

Strengths: One declaration can reach several providers, path-level rules are supported, and the design stays close to RFC 9309 so legacy parsers ignore it safely. Support has been announced by three major AI developers.

Limitations: It remains a draft with a six-month lifespan unless renewed, adoption dates stretch into 2027, the draft allows private agreements to override it, and it covers training only. The Verge reports Meta has not committed.

Outlook

The next steps are the vendors' own implementation updates, including OpenAI's promised follow-up, and the IETF working group's continued work on the broader vocabulary. Google's post notes that multistakeholder discussions continue on the rest of the AI preferences vocabulary, and Microsoft says work remains toward a final standard.

Conclusion

The proposal is a pragmatic, low-friction attempt to replace per-vendor training opt-outs with one robots.txt group. Its value depends on actual implementation, which none of the three named vendors had completed as of October 30 according to their own posts. Site owners can start planning now, while keeping existing crawler rules in place. Rating: 4 out of 5 for design clarity and breadth of announced support, tempered by draft status.

Editor's Verdict

Shared robots.txt Token Proposed for AI Training Opt-Outs earns a solid recommendation within the IT news space.

The strongest case for paying attention: one robots.txt declaration can reach several AI providers instead of per-vendor rules. That alone cuts the per-vendor maintenance site owners face today. Reinforcing that, path-level preferences are supported, such as opting out site-wide while allowing a press-kit directory — practical value rather than just headline appeal. The broader signal worth registering is straightforward: the mechanism is an Internet-Draft with Standards Track as its intended status, not a ratified standard, and it expires on 3 May 2027 unless renewed. On the other side of the ledger, one constraint is real rather than a marketing footnote: the proposal remains an Internet-Draft, and the IETF cautions against citing drafts other than as work in progress. It should factor into any serious decision. Layered on top of that, implementation dates stretch into 2027 for Google and Microsoft, and OpenAI gives only a coming-months window — which means no site is protected by the token alone yet.

For publishers, site operators, and SEO teams managing AI crawler policy, this is worth planning for now, while keeping existing per-vendor rules in place.