The bottom line
On October 30, Google, Microsoft, and OpenAI said they’ll honor a single robots.txt setting that tells AI companies not to train their models on your website. Instead of blocking each company’s crawler one by one, you add one rule for the shared token __usage_ai-training. It comes from an Internet Engineering Task Force (IETF) proposal, and support rolls out over the coming months and into 2027.
- OpenAI plans support “in the coming months”; Microsoft starts February 1, 2027; Google aims for 2027.
- Opting out of training is separate from appearing in AI search answers — you can do one without the other.
- Existing controls like Google-Extended and GPTBot keep working alongside the new setting.
- Most small businesses that want to be found in AI answers shouldn’t block AI search crawlers.
What was announced
Until now, a website that didn’t want its content used to train AI models had to block each company’s crawler by name, and keep that list current. On October 30, three of the biggest AI providers announced support for one shared signal instead:
| Company | What it said | When |
|---|---|---|
| Will treat the shared opt-out as if you’d blocked Google-Extended, its existing AI training control (announcement) | Aims to support it in 2027 | |
| Microsoft | Will honor the shared training setting and add a separate “grounding” setting for AI answers (announcement) | From February 1, 2027 |
| OpenAI | Will honor the common signal; its existing crawler controls still apply (announcement) | “In the coming months” |
The setting comes from an IETF draft published the same day, written by an author at Anthropic with contributors from Google, Anthropic, and major publishers. It builds on the IETF’s AI Preferences working group, which has spent more than a year defining what “AI training” means. Google notes the approach is consistent with the EU’s code of practice for AI model providers, which recognizes robots.txt as a main way for publishers to state their preferences. Honoring it is voluntary.
How the new setting works
Robots.txt is a plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers what they can access. The new setting uses a special name, __usage_ai-training, in the line where you’d normally name a crawler. A rule under it applies to every participating AI company at once.
Try this: the robots.txt rule
Microsoft’s announcement shows the whole-site version; the IETF draft shows how to carve out a folder:
# Opt the whole site out of AI training User-Agent: __usage_ai-training Disallow: / # Or opt out everything except one folder, like a press kit User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
A few details from the announcements:
- Allowing training is the default. If you add nothing, nothing changes.
- Company-specific rules still work. You can use the shared rule for a blanket choice and still set different rules for individual companies, like Google-Extended or OpenAI’s GPTBot.
- Microsoft adds page-level controls through a meta tag or HTTP header, and is retiring its older NoArchive and NoCache tags for this purpose. If your site uses them, Microsoft asks you to switch by March 1, 2027.
Training isn’t the same as AI search
This is the part that matters most for a small business. Training means using your content to build or update an AI model. Search and grounding mean pulling your page in when someone asks a question, often with a link back to you. The companies treat these separately:
| Company | Training control | AI search and answers |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot — OpenAI says you can allow it to appear in search results while blocking GPTBot from training |
| Anthropic | ClaudeBot | Claude-SearchBot and Claude-User — Anthropic says blocking them may reduce your visibility in search |
| Google-Extended (and the new shared setting) | Google says sites that allow Google-Extended are eligible to appear in Gemini app answers and receive clicks | |
| Microsoft | The new shared setting, plus page-level tags | A separate “grounding” setting; Microsoft warns that blocking it reduces your visibility in AI experiences like Copilot |
Watch out
Read Google’s detail carefully: if you use the new shared opt-out without separately allowing Google-Extended, Google will treat it as a Google-Extended block, and Google says sites that allow Google-Extended are the ones eligible to appear in Gemini app answers. If being cited by AI matters to you, decide on Google-Extended explicitly rather than relying on the blanket rule.
None of these settings affect regular search results. Microsoft says its preferences don’t change eligibility for Bing search, and none of the choices guarantee inclusion, traffic, or payment.
Should your business opt out of AI training?
| If your business… | Consider… |
|---|---|
| Sells its content — courses, research, paid articles, original designs | Opting out of training, while keeping AI search allowed so people can still find you |
| Depends on being found — local services, retail, restaurants, B2B | Leaving AI search and grounding open; the training choice matters less than visibility |
| Isn’t sure | Changing nothing yet; allowing training is the default, and support doesn’t start until the coming months |
| Uses Microsoft’s NoArchive or NoCache tags | Moving to the new settings before March 1, 2027 |
Two limits to keep in mind. Robots.txt is a request that participating companies agree to honor, not a lock. And it applies going forward: it signals what future crawls shouldn’t be used for, rather than removing content from models already trained. If you’re still getting your bearings on how these models work, see what generative AI is.
Your next step
What to do now
- Open yoursite.com/robots.txt and see which AI crawlers, if any, you already block.
- Decide separately whether you want to allow AI training and whether you want to appear in AI answers.
- If you opt out of training, add the shared rule — and set Google-Extended, GPTBot, and others explicitly if you want different choices per company.
- If you use NoArchive or NoCache for AI reasons, plan the switch before March 1, 2027.
- Check back in 2027, when Google and Microsoft support goes live.
Keep going
Business Prompted publishes practical guides, tool reviews, and workflows for people who run businesses. Explore more at businessprompted.com.
Common questions
How do I opt my website out of AI training?
Add a robots.txt group with the user agent __usage_ai-training and Disallow: /. Google, Microsoft, and OpenAI say they’ll honor it, with support arriving in the coming months and into 2027.
Will opting out of AI training remove my site from Google search?
No. These preferences don’t affect regular search results. Google and Microsoft both say traditional search eligibility is separate.
Will it remove my site from ChatGPT or Gemini answers?
Not by itself — training and AI search use separate controls. But Google says sites that allow Google-Extended are the ones eligible to appear in Gemini app answers, so decide on Google-Extended explicitly.
When does the new setting take effect?
OpenAI says in the coming months, Microsoft from February 1, 2027, and Google aims for 2027.
Do I still need to block GPTBot or Google-Extended separately?
Not if you want the same choice everywhere; the shared rule covers participating providers. Company-specific rules still work if you want different choices for different companies.
Does Anthropic honor the new setting?
The IETF draft was written by an author at Anthropic, but Anthropic hadn’t published its own timeline when we checked. Its existing ClaudeBot controls continue to work.
Sources & further reading
- Google Search Central — Supporting a shared AI training control for websites (Oct 30, 2026)
- Microsoft Bing — Your content, your choice: clearer preferences for training and grounding (Oct 30, 2026)
- OpenAI — Making it easier for websites to express AI training preferences (Oct 30, 2026)
- IETF — Machine-Readable Training Preference Signal (draft, Oct 30, 2026)
- OpenAI — Overview of OpenAI crawlers
- Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
Business Prompted guides are written for practical use and checked against the sources listed above. Read our editorial policy.
