Latest in AI
OpenAI Backs a Shared AI Training Opt-Out Signal in robots.txt
A proposed common signal could simplify publishers' controls, but it does not replace crawler rules or arrive immediately.
OpenAI said on October 30 that it plans to honor a shared robots.txt signal for AI training preferences. Website owners could express one choice across participating providers rather than maintain a separate rule for each. Implementation is expected in the coming months, not today.
That timing matters if you run a publication, documentation site or company blog. This is a commitment to support a common mechanism, not a reason to remove the controls you already use.
What OpenAI announced
The announcement describes a voluntary effort connected to the IETF AI Preferences working group's discussions. OpenAI says its existing crawler controls will remain in place and that it will publish implementation details later.
The useful change is administrative. A publisher's preference should not require a growing inventory of every AI company's crawler name. A shared signal could make that preference easier to maintain, although its practical reach still depends on which providers honor it.
How the proposed training signal works
The linked Internet-Draft uses a purpose-specific token inside the familiar robots.txt format:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
In that example, the preference is against training on the site except for the press-kit path. The draft defines training around changing a generative model's learned parameters. It separates that use preference from rules governing crawler traffic.
Expressing a training preference is different from blocking a crawler. The proposed token is not itself a command to stop fetching pages. A site can state how it wants content used while separately managing which bots may retrieve it.
This is still an Internet-Draft, not a finished IETF standard. The example explains the proposal OpenAI linked, rather than guaranteeing a final implementation or support across every AI provider.
Can you block training without leaving ChatGPT search?
OpenAI's existing crawler documentation already separates those choices. GPTBot relates to content that may be used for foundation-model training; OAI-SearchBot controls inclusion in ChatGPT search results. Website operators can allow the latter while disallowing the former.
ChatGPT-User is a separate user agent for user-triggered visits. OpenAI says robots.txt rules may not apply to those visits because a person initiated them. Treating every OpenAI request as the same kind of crawling would miss that distinction.
For now, the practical step is to review those existing settings against what you actually want: search discovery, training use, or restrictions on automated access. The new announcement does not make those decisions interchangeable.
The part worth watching
A common syntax can reduce configuration work, but syntax alone cannot tell a publisher how consistently a preference is followed. The next useful evidence will be providers' implementation details and a clear account of which uses each control covers.
For publishers, the important outcome is being able to make a specific choice without accidentally making a different one. Keeping search discovery, training permission and crawler traffic separate is what makes that possible.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help technical founders explain product changes clearly and turn that thinking into content developers and founders want to read.