On 30 October 2026 Google said it plans to support a shared Google AI training control in robots.txt, a single policy token that site owners can use instead of writing separate rules for each AI company. It is not live yet: Google’s crawler documentation says early 2027. Your existing Google-Extended rules keep working, so for now the job is to check your robots.txt, not to rewrite it.
If you run a WordPress, WooCommerce or Shopify site, you have probably seen the advice to block AI crawlers one by one. Every AI company has its own user agent, and your robots.txt file grows with each new name. The Google AI training control in robots.txt is an attempt to fix that with one shared signal.
Here is what Google actually announced, what it does not change, and the short checklist worth running this week. Everything below comes from Google’s own blog and documentation, which we opened and checked in November 2026.
What Google announced on 30 October 2026
Google published “Supporting a shared AI training control for websites” on the Search Central blog on Friday 30 October 2026. It was written by John Mueller. The post says Google plans to support a shared control in robots.txt, based on work from the IETF’s AIPREF working group. Site owners would use one “policy token” to set a broad training preference across participating providers.
Three details matter more than the headline:
- It is a plan, not a launch. Google says it aims to support the control alongside other AI providers in 2027, and its crawler documentation says “early 2027”.
- Google-Extended stays. Google says the existing company-specific controls “will continue to function alongside this new joint signal”.
- The token is a user-agent name. The documentation shows it as
user-agent: __usage_ai-training, so it goes in an ordinary robots.txt group and adds no new directives.
How the Google AI training control in robots.txt works
Today, if you want to stop Google from using your content to train its Gemini models, you disallow the Google-Extended token. Google describes it as a standalone product token that does not affect your inclusion in Google Search and is not a ranking signal.
The new token sits on top of that. Per the announcement, if a site uses the shared token to opt out of training and has not explicitly set a Google-Extended rule, Google plans to treat it as if the site had disallowed Google-Extended. In other words, the shared token only stands in when Google-Extended is not explicitly specified. Google also notes that sites which explicitly allow Google-Extended can still appear in Gemini Apps responses and receive clicks through grounding.
The proposal behind the token is an IETF Internet-Draft called “Machine-Readable Training Preference Signal”, dated 30 October 2026. It is an individual submission, the document itself says it is not endorsed by the IETF, and it expires on 3 May 2027 unless it is revised. The listed author works at Anthropic, which we mention so you can weigh the source for yourself. It proposes that a Disallow rule in the token’s group signals a preference against AI training and an Allow rule signals a preference in favour.
| Control | What it does | Status in November 2026 |
|---|---|---|
Google-Extended |
Manages whether Google-crawled content may train future Gemini models. Does not affect Search inclusion. | Live. Google has offered it since 2023. |
__usage_ai-training |
One shared policy token for AI training preferences across participating providers. | Planned. Google says early 2027. |
Rules for Googlebot |
Control crawling for Google Search. | Unchanged by this announcement. |
noindex or a password |
Keep a page out of Google Search or behind a login. | Unchanged. robots.txt is not a way to do this. |
Should you change your robots.txt today?
No rush. Google does not read the new token yet, so adding it now does nothing for Google. What you should do is find out what your file says today, because an old rule can surprise you later. The two things to look for:
- A
Google-Extendedgroup you added a while ago, whether it allows or disallows. Google says the shared token only applies when Google-Extended is not explicitly specified, so any explicit rule should match what you want now. - A blanket rule copied from a “block all AI bots” tutorial that you never reviewed. Make sure you know which crawlers it names and why.
Blocking training is a business choice, not a technical one. Google notes that sites which explicitly allow Google-Extended can still appear in Gemini Apps responses, so weigh that before you opt out. Decide on purpose, then write the rule.
How to check and edit robots.txt in WordPress
WordPress generates a virtual robots.txt when no physical file exists. Its default output is a User-agent: * group that disallows the admin area and allows admin-ajax.php. SEO plugins and hosts often replace or extend it, so always check the live file first.
- Open
yoursite.com/robots.txtin a browser and copy what you see. - Look for
Google-Extended,User-agent: *groups and anyDisallow: /line you do not recognise. - If you use Rank Math, go to Rank Math SEO, turn on Advanced Mode, open General Settings and click Edit robots.txt.
- If a physical robots.txt file sits in your site root, Rank Math’s guide says to delete it first, otherwise the physical file may take precedence. Back it up before you do.
- To opt out of Gemini training today, add a group with
User-agent: Google-ExtendedandDisallow: /, then save. - Reload
/robots.txtand confirm the change is live. If you use a cache or a CDN, purge it.
Add your changes as a separate group. Do not edit the existing User-agent: * group, which protects /wp-admin/ and keeps admin-ajax.php reachable.
If you want to confirm that Google can still crawl your pages after any change, our guide to checking whether Googlebot can crawl a page walks through five free tests.
What robots.txt will not do
Google’s own robots.txt introduction is blunt about the limits, and they apply to the new token too:
- It is not a way to keep a page out of Google. Use
noindexor a password for that. - It is not access control. Crawlers choose whether to obey it, and some do not.
- A blocked URL can still show up in results without a description if other sites link to it.
A shared training signal is also a preference that depends on each company honouring it. If your traffic moved recently, check Google’s ranking updates before blaming robots.txt, starting with our note on the September 2026 spam update.
Nothing breaks and nothing is required. Read your current robots.txt, make sure any Google-Extended rule says what you want, and wait for Google to document support for the shared token. When it does, add the new group and keep your per-company lines.
Frequently asked questions
Is the shared AI training control live in Google right now?
No. Google’s blog post says it aims to support it in 2027, and its crawler documentation says early 2027. Until then, Google-Extended is the control that works.
Will blocking Google-Extended hurt my Google Search rankings?
Google says Google-Extended does not impact a site’s inclusion in Google Search and is not a ranking signal.
Do I need to remove my Google-Extended rule when the new token arrives?
No. Google says existing company-specific controls will keep working alongside the shared signal.
Is the __usage_ai-training token an official web standard?
Not yet. It comes from an individual IETF Internet-Draft that the IETF has not endorsed, and the draft expires on 3 May 2027 unless it is revised.
Sources: Google Search Central Blog; Google common crawlers; IETF draft; Google robots.txt introduction; WordPress do_robots(); Rank Math robots.txt guide. Checked November 2026.
More in Software & Tools
WordPress Reprint Plugin: What It Does and How to Use It
WordPress.org's new Reprint plugin copies a site's files and database to a laptop or new host. See requirements, setup steps and the limits to know first.
Google November 2026 Core Update: What to Check Now
Google's November 2026 core update began on 10 November and may take two weeks. Here is what to check, when to wait, and what not to change.
WooCommerce 11.2.2 Security Update: What to Check Now
WooCommerce 11.2.2 security update: what the 4 November release hardens, the database update, and a safe step-by-step way to update your store.