By the Mayab Media Team
For two years, keeping your content out of AI training meant chasing crawler names. GPTBot, ClaudeBot, Google-Extended, CCBot. Every few months another one showed up, and your robots.txt file grew another block that nobody on the team remembered adding.
On October 30, the big players agreed on a shortcut.
Google, Microsoft, OpenAI and Anthropic each said they will respect a single shared token in robots.txt: __usage_ai-training. You put it where a crawler name normally goes, add a Disallow line, and that's your answer to every company that honors it. It comes out of a proposal in the IETF's AI preferences group, written by an Anthropic engineer.
Sounds tidy, and it mostly is. But the timing is staggered, the scope is narrower than people assume, and one detail at Google could cost you visibility you actually want.
What the line does (and when it starts working)
Here's the whole thing for a site that wants out of training but is fine sharing a press page:
User-agent: __usage_ai-training
Disallow: /
Allow: /press/
That's a preference about how your content gets used to train generative models. It isn't a crawl rule, so bots can still visit your pages.
Nobody flips the switch on the same day, either, so expect a messy year. Google's crawler documentation says support arrives in early 2027. Microsoft starts honoring it on February 1, 2027. OpenAI says "in the coming months," with no date. Anthropic says it has already started implementing it.
One quirk caught our eye. A blanket User-agent: * block doesn't count as a training preference under the draft. If you assumed your catch-all rule covered AI training, it doesn't.
And the IETF itself says the draft has no formal standing yet, because it's still an individual submission. Wording could change.
The Google catch nobody's talking about
This is the part we'd flag to any client before they touch the file.
At Google, the shared token maps onto Google-Extended. If you opt out with it and don't mention Google-Extended separately, Google will treat your site as if you'd blocked Google-Extended. That control covers more than model training. It also governs grounding in Gemini Apps, meaning whether Gemini can pull your pages into an answer and link to you.
Say you run a dental clinic with two locations and your web person adds the line to "protect the content." You may also be quietly stepping out of the answers Gemini gives when someone asks for a dentist nearby. Probably not what you had in mind.
What it doesn't touch: regular Google Search, AI Overviews and AI Mode. Those still run on Googlebot rules and the separate Search Console controls. Your rankings won't move.
Bing went further (and set deadlines)
Microsoft split things into two separate dials. AI Training is one. AI Grounding is the other, and you set it page by page with a meta tag or an X-Robots-Tag header rather than in robots.txt.
Grounding has three settings. Allow is the default. DisallowExtended keeps you in Microsoft's own products (Copilot, Bing's AI answers, Microsoft 365) but out of third-party services built on Bing data. Disallow removes you from AI answers entirely, and Microsoft warns plainly that this cuts brand visibility.
There's a cleanup job hidden here, too. Some sites use NoArchive or NoCache tags, sometimes because an old SEO plugin added them. After March 1, 2027, Bing stops reading those as AI preferences. If you relied on them, you'll need the new settings by then. A temporary grounding option opens in Bing Webmaster Tools on December 15.
What we'd actually do
For most of the businesses we work with, the answer is simple: leave AI training allowed and don't add the token yet.
A plumber, a law firm or a software company wants to be the name an assistant recommends when a buyer asks for options. Blocking training doesn't stop competitors from being mentioned. It just takes you out of the conversation (at Gemini, literally).
The math is different for publishers, course creators and anyone whose content is the product. If people pay for your articles, keeping them out of training makes sense. Even then, we'd use Bing's split controls to block training while keeping grounding, so you still collect the citations and the clicks.
Before February, a quick checklist:
- Open your live robots.txt (yoursite.com/robots.txt) and see what's already there. Plenty of sites have AI bot blocks they forgot about.
- Search your page templates for noarchive or nocache. If they're there by accident, remove them.
- If you're on a hosted builder, look in its crawler or SEO settings. Some now offer an AI-crawler toggle that may not map cleanly to the new token.
- Write down your decision. When Google's support goes live in 2027, you'll want to know why the file says what it says.
One more caveat. This is a preference, not a lock. Nothing in the draft forces any company to comply, and Meta, Amazon and Apple made no commitments in these announcements.
Pick the line deliberately, because at Google opting out of training can also mean opting out of being cited.
Want us to check how this affects your business? Book a free audit.
