One Robots.txt Rule to Refuse AI Training: What Big Tech Agreed On

November 1, 2026 · Cortex Solutions

Google, Microsoft, OpenAI and Anthropic backed one robots.txt signal for refusing AI training. Here is what it does, what it does not, and what Saudi firms should do.

On October 30, 2026, four of the biggest names in artificial intelligence moved in one direction. Microsoft, Google, OpenAI and Anthropic said they will support a shared signal that lets any website owner refuse the use of their content for training AI models, through a single rule in the familiar robots.txt file. According to SiliconANGLE's coverage that same day, and a PPC Land report dated October 31, one rule can now address these companies together.

The idea sounds simple, but the details matter. This is a declared preference, not a lock, and the rollout dates differ from one company to another.

What exactly was announced?

The signal rests on a technical draft published with the Internet Engineering Task Force (IETF) on October 30, 2026 under the title Machine-Readable Training Preference Signal. Its sole author is Kevin Kelley of Anthropic. It defines a new token called __usage_ai-training, which is written in robots.txt in the place normally used for a crawler's name.

According to the examples in Microsoft's post on the Bing blog, dated October 30, a site owner refuses training across the whole site like this:

User-Agent: __usage_ai-training

Disallow: /

A single path can be excluded by writing it instead, such as Disallow: /example-path/ and Microsoft notes that allowing is the default. The draft's own example also shows that a site can refuse training overall while allowing one folder, such as a press kit.

When does it take effect?

The four companies did not commit to one date:

• Microsoft: its Bing post states that it will honor the new preferences as it crawls from February 1, 2027.
• Google: according to PPC Land, quoting Google's blog and crawler documentation, it aims to support the signal in 2027, with the documentation saying early 2027.
• OpenAI: PPC Land quotes it as aiming to implement the signal in the coming months, with no specific date.
• Anthropic: Kevin Kelley wrote on the working group's mailing list, as quoted by PPC Land, that the company has already started implementing the draft, and that adding support across its services will take some time.

What does this signal not do?

This is the most important part for anyone deciding how to use it:

• It does not block access. The draft says plainly that the token is not a directive for crawling behavior, and Bing says these settings govern the use of content, not crawl access.
• It has no enforcement mechanism. The draft has not been endorsed by the IETF and carries no formal standing in its standards process, and it expires on May 3, 2027 unless updated.
• It does not bind everyone. SiliconANGLE reports that Meta expressed support for the idea but has not yet committed to implementing it.
• It does not cover everything. The draft excludes models used exclusively for non-generative tasks, such as classification and ranking, and says private agreements may override these preferences.
• It may not reach past collection. The draft ties preferences to the file in effect when the content was retrieved, and AI Weekly reports that the signal does not remove content already in existing training sets.

There is also a distinction between training and grounding, meaning a model fetching a page to answer a user's question. Microsoft says the training preference does not cover this use, and is adding a separate AI Grounding preference set at page level, not site-wide, stressing that neither choice guarantees inclusion, traffic or payment. Google differs: PPC Land reads its documents as treating the shared opt-out like a Google-Extended refusal, which covers training and Gemini grounding alike.

The new signal also does not cancel what came before. According to PPC Land, OpenAI said it will not replace the controls it already provides through robots.txt, and Kelley said it complements Anthropic's existing crawler controls.

What does this mean for Saudi businesses?

Any Saudi company with a website, online store or knowledge base will have a clear, shared tool for stating its position, and it is worth deciding that position now rather than leaving it to defaults, since allowing is the default:

• Sort your content. Separate what you want to spread widely, such as product descriptions and press material, from what took real effort to build, such as guides and specialist research.
• Separate training from visibility. Microsoft says these preferences do not change eligibility for traditional Bing search results, so refusing training does not mean disappearing from them. Read each search engine's terms before you apply the rule.
• Keep your current rules. Add the new token alongside any crawler-specific rules you already use, and do not replace them.
• Do not treat it as protection. Sensitive or customer data should never rely on robots.txt. A preference is a request to those who choose to honor it, and data that must stay private belongs behind a login.
• Review your contracts. Since private agreements may override these preferences, check what your agreements with content platforms and technology vendors allow.

The rule itself takes minutes to add, but the decision behind it deserves a short conversation between management and marketing. The content of your website is an asset, and now there is a simple, shared way to say how you want it used.