The short version
- Google-Extended is a robots.txt setting that asks Google not to use a site for Gemini; 12.3% of top sites with a readable robots.txt use it to block their whole site (our crawler study, September 2026).
- Across 11,500 queries collected in December 2025, sites blocking Google-Extended were significantly less likely to be cited in AI Overviews, and 21 popular publishers that block it were never cited by Gemini (Grossman and colleagues).
- In our same-day check of 481 AI Overviews in September 2026, 17.8% of cited pages on top sites came from sites using Google-Extended, close to the 16.8% expected, so no clear effect showed.
- Training and search blocks are separate elsewhere too: 51.7% of sites blocking OpenAI’s training crawler still allowed its search crawler, and ChatGPT cited none of 123 checked pages closed to that search crawler.
- Blocked AI agents answer anyway from other sources: in a 2026 vendor test, answers drew 58% from a business’s own pages when its site was hard to read, against 78% when it was easy.
What does Google-Extended actually control?
It controls whether Gemini may use your content, not your Google ranking. According to Google’s own description, as reported by Grossman and colleagues (opens in a new tab), sites can block it to keep their content out of training future Gemini models and out of the sources Gemini cites. Google also says it does not affect ranking in Google Search, nor AI Overviews’ access to the content.
Google-Extended is a name that robots.txt files can address, not a separate crawler. In our crawler blocking study of 5,572 top sites with a readable robots.txt, 12.3% blocked Google-Extended from the whole site. 10.1% of all sites opted out of Gemini training this way while still allowing Googlebot, Google’s search crawler.
The only way to keep content out of AI Overviews, during spring 2026, was through general search settings. Xu and colleagues (opens in a new tab) note that publishers could restrict AI Overview use with noindex and nosnippet instructions, which also affect regular search results.
Does blocking it reduce AI Overview citations?
One large study says yes; our smaller check found no clear difference. Grossman and colleagues collected Google results, AI Overviews and Gemini answers for 11,500 queries, collected in December 2025 from a single location in New Jersey. AI Overviews appeared on 51.5% of the representative real-user queries.
Comparing each source’s chances in AI Overviews with its chances in the regular results for the same query, they found that sites blocking Google-Extended were significantly less likely to be cited in AI Overviews. Sites such as the New York Times, Yelp, IMDb and Facebook had fewer AI Overview citations. The authors call this surprising and suggest publishers may need to rethink the block.
Our cross-check pointed the other way. We matched the sources in 481 US AI Overviews from 26 September 2026 against robots.txt files. 17.8% of cited pages on top-10,000 sites came from sites using Google-Extended, against 16.8% expected from their popularity.
| Study | When | What it found |
|---|---|---|
| Grossman et al., 11,500 queries | December 2025 | Blocking sites significantly less likely to be cited in AI Overviews |
| Underneath, 481 AI Overviews | September 2026 | 17.8% of cited pages from blocking sites, against 16.8% expected |
Neither study is a controlled test. Sites that block Google-Extended differ from other sites in size and type, and both studies cover a single point in time.
What happens to your visibility in Gemini?
Gemini stops citing you, as Google says it will. Grossman and colleagues found 21 popular publishers, each retrieved for at least 20 queries by both Google Search and AI Overviews, that Gemini never cited. All of them block Google-Extended.
The list included major news and reference brands. Popular social and review sites, including Facebook, Instagram, TikTok, IMDb, Yelp and Tripadvisor, also received no Gemini citations. The authors describe the resulting loss of Gemini visibility as self-inflicted, because Google is doing what these sites asked.
Robots.txt is not the only factor, though. In our study, all 64 AI Overview citations of pages closed even to Googlebot were Reddit pages, and Google has a data licensing agreement with Reddit.
Do other AI companies treat training and search separately?
Yes. Most run separate crawlers, and the search crawler is the one tied to citations. In our crawler study, 51.7% of the sites that blocked OpenAI’s training crawler, GPTBot, still allowed OAI-SearchBot, the crawler behind ChatGPT search.
On the same day, ChatGPT cited 123 pages on top sites whose robots.txt we could check, and none was closed to OAI-SearchBot. Six of them were closed to GPTBot and were cited anyway. In that sample, training opt-outs did not keep pages out of ChatGPT answers, while search opt-outs did.
Some blocks are accidental. 3.8% of sites block every unnamed bot from the whole site, which also shuts out AI crawlers that did not exist when the rule was written.
What happens when an AI agent cannot read your site?
It answers anyway, using other people’s pages. Finder and colleagues (opens in a new tab) ran buyer questions about 1,056 businesses through four AI agents. In about 99% of journeys that hit a dead end on a site, the agent still answered, from whatever it found elsewhere.
When a site was easy for agents to read, 78% of the answer came from the business’s own pages. When it was not, that fell to 58%, and web search filled the gap. Two independent AI judges both rated the answer a clear recommendation 20% of the time for easy-to-read sites, against 11% for the rest. We cover that study in our guide on agent-readable websites.
This is a different setting from Google-Extended, and the authors work for a company that sells readiness scores. It still shows the general trade-off: an AI that cannot use your content does not stay silent about you. Whether outside coverage makes up for it is covered in our guide on off-site mentions and readability.
What should you do about it?
Make the Google-Extended decision deliberately, with the trade-off written down. Practical steps:
- Decide what you are protecting. Blocking Google-Extended keeps content out of Gemini training and Gemini citations; it may also cost AI Overview citations.
- Audit robots.txt crawler by crawler. Separate training crawlers, AI search crawlers and fetchers that act for a user, and check for old rules that block every bot.
- Do not use noindex or nosnippet to avoid AI Overviews unless you accept losing regular search visibility too.
- If you block or unblock Google-Extended, track AI Overview and Gemini citations for your key searches before and after the change.
- Check your firewall and content delivery network as well, since many blocks happen there rather than in robots.txt. Your server logs can also show which pages AI bots request.
If you want an outside view of these settings, our generative engine optimization service covers crawler access.
What does the research not tell us yet?
Whether the block itself causes fewer AI Overview citations is still unproven.
- Both studies are observational snapshots. No one has yet compared the same sites before and after changing the setting.
- The two results disagree, and they differ in date, sample size and method.
- The Grossman study collected data from one location on mobile; our check covered a few hundred AI Overviews on one day.
- Google can change how AI Overviews treat Google-Extended at any time, so findings may date quickly.
- Robots.txt shows what a site asks, not what crawlers do; blocks at the firewall were not measured.
Frequently asked questions
Does blocking Google-Extended affect my Google search rankings?
Google says it does not. Google-Extended governs Gemini training and Gemini citations; our study found 10.1% of top sites block it while still allowing Googlebot.
Can I opt out of AI Overviews without leaving Google search?
Not cleanly. During spring 2026, the available controls were noindex and nosnippet, which also affect regular search results, according to Xu and colleagues.
Should we unblock Google-Extended?
It depends on what you value more. A large December 2025 study linked the block to fewer AI Overview citations, while our September 2026 check saw 17.8% of cited pages from blocking sites against 16.8% expected.
How many websites block Google-Extended?
Among top sites with a readable robots.txt, 12.3% blocked it from the whole site in September 2026.
Sources
- Grossman, Liu, Chen, Smith, Borcea and Chen (2026), How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews (opens in a new tab), arXiv:2604.27790.
- Xu, Iqbal and Montgomery (2026), Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact (opens in a new tab), arXiv:2605.14021.
- Finder, Elovic, Shalev and Yosef (2026), AX is the New AEO (opens in a new tab), arXiv:2609.34951.
- Underneath (2026), Which AI crawlers do top websites block? 10,000 sites, 2026