Which AI crawlers do top websites block?
The robots.txt of the top 10,000 sites: 15.2% block GPTBot, and half of those still allow ChatGPT’s search crawler.
Underneath Research
Research
Original data on how AI finds and cites.
Twenty-three studies of AI crawlers, Google AI Overviews and AI Mode, the brands ChatGPT, Gemini, Perplexity and Claude recommend, and what they say about businesses. Every study publishes its method, its limitations and the per-item data behind each number, free to reuse with attribution.
ChatGPT local recommendations vs Google Maps
We compared 1,405 local business listings from ChatGPT in the United States, United Kingdom, Canada and Australia with Google Maps. For the same business at the same location, 99.7% of names, 98.1% of website links and 99.6% of phone numbers were identical, including tracking tags that exist only on Google Business Profiles.
Research · Local AI search
Twenty-three studies, all the data published.
Which Google Maps businesses does ChatGPT recommend?
898 ChatGPT answers over two days: more reviews than the local median raised a Maps business’s chance of being listed by about 20 points.
Underneath Research
How many websites have an llms.txt file?
11.5% of live top sites serve a valid llms.txt; more return an HTML page at the address instead.
Underneath Research
Do websites serve Markdown to AI agents?
3.2% of top sites answer an agent’s request for Markdown, with pages a median 96.1% smaller than the HTML.
Underneath Research
When does Google show an AI Overview?
1,248 US searches: local maps, wording and query mix explain more than industry; most cited sources do not rank on page one.
Underneath Research
Do AI Overviews cite the pages that rank?
4,051 citations: only 28.7% rank in the top 10 for the same search, and YouTube appears in 64.2% of AI Overviews.
Underneath Research
AI Mode vs AI Overviews: how different are the sources?
For the same search, Google’s two AI answers share a mean 13.9% of cited URLs.
Underneath Research
What pages cited by AI Overviews have in common
Among top-10 pages on the same search, ranking position matters far more than schema or other markup.
Underneath Research
How many “best of” lists cited by AI rank their own brand first?
269 numbered lists cited by six AI surfaces: 24.2% were published by a company that ranked itself first.
Underneath Research
The YouTube videos Google’s AI cites are small
469 videos cited in AI Overviews and AI Mode: a median of 4,283 views, and 60.9% under 10,000.
Underneath Research
When does a Reddit thread become evidence in Google’s AI?
17.9% of AI Overviews cite Reddit. Cited threads have more discussion than the ones Google shows and the AI skips, and a fifth of the sentences citing them are not supported.
Underneath Research
How fresh are the pages AI engines cite?
Against Google’s top 10 for the same questions, AI assistants cited pages first published about half as long ago.
Underneath Research
Do ChatGPT, Gemini, Perplexity and Claude agree on brands?
80 buyer questions: all four assistants recommended the same first pick for 10.0% of questions.
Underneath Research
Ask an AI the same question 5 times: do the brands change?
About a quarter of the brands ChatGPT named appeared in all five answers to the same question, and one answer showed about half of them.
Underneath Research
Same question, four countries: do AI recommendations change?
Asked five times in each of four countries, ChatGPT’s brands overlapped 0.594 within a country and 0.429 across countries.
Underneath Research
Does rewording a question change AI brand recommendations?
Adding “on a tight budget” kept the same first brand 15.3% of the time across three engines, against 68.0% for the same question asked again.
Underneath Research
The hidden searches AI assistants run before they answer
ChatGPT ran 3.7 searches per answer, and added a year to them in 52.5% of answers.
Underneath Research
Do ChatGPT, Gemini, Perplexity and Claude cite pages that rank?
Only 8.3% of ChatGPT’s citations ranked in Google’s top 10 for the same question.
Underneath Research
How faithfully do AI assistants quote software prices?
61.9% of 840 quoted plan prices were fully faithful to the pricing page; most differing prices still appear on the vendor’s own site.
Underneath Research
“Is this brand legit?” How AI assistants build a reputation
79 brands, four engines: every complete answer called the brand legitimate, and 99.7% then raised a problem.
Underneath Research
Do AI answers match a business’s Google profile?
159 local businesses, four engines: 18.9% of answers stated a fact that differed from the business’s Google profile, often one the business publishes elsewhere.
Underneath Research
Do Wikipedia and schema make AI assistants recommend a brand?
Brands on Wikipedia are named by more assistants, but within the same question most of the gap goes once brand prominence is taken into account.
Underneath Research
ChatGPT local recommendations vs Google Maps
1,405 listings in four countries matched against Google Maps, field by field.
Underneath Research
Every number traces to data.
Each study states its question, sample, method and limitations, and publishes the per-item data behind every figure. Every statistic is computed directly from that dataset, so anyone who downloads it can reproduce the number, and each figure in the text is checked against the data before publication. Comparisons with other studies quote the original publisher. Corrections are logged in each study’s changelog.
Free strategy call
Want these numbers for your own category?
We run the same measurements for a business’s own buyer questions and competitors. On a free 30-minute call we’ll take a first look and send you a short written read afterward.