• AI visibility

Guide · AI search

Can our server logs show what content AI bots are looking for on our site?

Partly: your server logs record every page AI bots request, including addresses that do not exist, and those failed requests can hint at content people are asking AI assistants about. What logs cannot show is whether a page an AI bot read was used in an answer, cited, or seen by anyone. One published case used AI bot 404 errors to decide which pages to create, but it changed several things at once, so the tactic’s own effect is unknown.

By Underneath Team Guide Updated

The short version

  1. One website treated AI bot requests for missing pages as demand signals and built new pages from the most frequent patterns; the study could not separate that tactic from three others applied at the same time.
  2. On that site, ChatGPT referrals to changed pages grew 6.1 times in four months, but untouched pages grew 3.5 times, and the best estimate of the real effect, about 1.8 to 2.3 times, was only suggestive.
  3. A bot visit is not a citation: when ChatGPT’s developer interface searched for shopping answers it returned 37.38 pages on average but cited 3.22.
  4. Bots make 62.4% of requests for web pages on one major network, so AI demand signals sit inside a lot of other bot traffic.
  5. Robots.txt rules do not map neatly onto citations: Perplexity cited 209 of 517 pages (40.4%) that sites had closed to its crawler.

What can AI bot requests in our logs tell us?

They show which pages AI systems ask for, how often, and which addresses they request that do not exist.

The volume is large and mixed. A 2026 study by Finder and colleagues at ora (opens in a new tab), an agent-readiness vendor, cites Cloudflare data showing bots made 62.4% of requests for web pages on its network in September 2026. Much of that is old search machinery. The fastest-growing part is AI agents that fetch a page because a person has just asked a question.

Those bots come in different kinds. Our crawler study separates training crawlers such as GPTBot, search crawlers such as OAI-SearchBot, and fetchers such as ChatGPT-User that load a page when someone asks. Sites treat them differently: 15.2% of top sites block GPTBot, but only 7.4% block OAI-SearchBot. Reading your logs by bot type tells you whether a request feeds a future model, a search index or a live answer.

Has anyone used AI bot errors to decide what pages to create?

Yes, one published case did, as part of a wider set of changes on a single website.

Watanabe and Nakayashiki (opens in a new tab) work at Glasp, a site with hundreds of thousands of question-and-answer pages about YouTube videos. In January 2026 they treated AI bot requests for missing pages as demand: “the highest-frequency patterns were used to generate new pages.” They also merged duplicate addresses, rewrote titles as questions, and ranked pages for rewriting by how often AI bots requested them.

A fourth rule protected search traffic. Pages already earning Google clicks were locked against rewriting. The authors are explicit that the four changes were made together, so they estimate “its combined effect, not the marginal effect of any single tactic.”

Did acting on those log signals increase AI traffic?

Possibly, but the gain was far smaller than the raw numbers suggest and did not pass the strictest test.

The raw growth was striking. ChatGPT referrals to the changed pages grew 6.1 times between January and May 2026. But pages the team never touched grew 3.5 times over the same months, because ChatGPT itself was growing. Comparing the two groups week by week, the authors estimate the changes lifted referrals by about 1.8 to 2.3 times.

Even that estimate is labelled suggestive. When they checked whether similar jumps happened by chance before the changes, the chance came out at 0.16, too high to rule out luck. Google clicks to the changed pages fell about 25%, close to the 20% fall across the whole site, so the rewrites did not obviously hurt search.

What the Glasp data showsFigure
Growth in ChatGPT referrals, changed pages6.1 times
Growth in ChatGPT referrals, untouched pages3.5 times
Estimated effect of all four changes togetherabout 1.8 to 2.3 times
Effect of the 404-based pages alonenot measured

What can’t server logs tell us?

They show what a bot read, not whether an answer used it, cited it or reached a buyer.

The ora team, who logged 37,927 AI agent journeys across 1,056 businesses, put it simply: “Logs show what the agent read, not what the answer used.” Uberti-Bona Marin and colleagues (opens in a new tab) measured the gap for ChatGPT’s developer interface on shopping questions. Its searches returned 37.38 pages per answer, but only 3.22 appeared as citations. Just 27.5% of the websites it found made it into the final answer.

The request itself is also one step removed from the buyer. In our hidden searches study, ChatGPT ran 3.7 searches per answer on average, and none of 509 searches repeated the user’s question word for word. A bot’s request reflects the assistant’s own rewritten search, not the buyer’s words.

Logs can also miss citations entirely. On the day of our crawler study, Perplexity cited 209 of 517 pages (40.4%) on top sites that had closed those pages to its crawler. So the crawler named in your rules is not the only route by which a page reaches an answer.

Do agent-facing files like llms.txt show up as demand?

Rarely, by the evidence so far, so a quiet llms.txt in your logs is normal rather than a warning sign.

Our llms.txt study reports that Ahrefs found 97% of the llms.txt files it monitors received no traffic in May 2026, and AI retrieval bots made 1.1% of the requests that did arrive. The ora authors report that only one in three agents that read an llms.txt went on to fetch a page it listed. Few sites offer agents a lighter format either: 3.2% of top sites return Markdown when an agent asks for it, in our agent-readable web study.

What should you do about it?

Use AI bot logs as a cheap source of content ideas, and test any pages you build against a control.

  1. Separate AI bot traffic by type: training crawlers, search crawlers and fetchers that act for a person.
  2. Pull the missing addresses AI bots request most often and group them into topics. Treat them as hypotheses about buyer questions, not proof of demand.
  3. Check those topics against what assistants actually answer for your buyers’ questions before building pages.
  4. When you add or rewrite pages, leave a similar group untouched, as Glasp did, so platform growth is not mistaken for your effect.
  5. Protect pages that already earn search traffic before rewriting anything.
  6. Make sure your bot rules do not block the fetchers that act for buyers, unless you mean to. Weigh what blocking Google-Extended may cost as well.

If you want help turning log signals into a measured content program, see our generative engine optimization service.

What does the research not tell us yet?

It has not tested whether pages built from AI bot 404 errors earn citations or traffic on their own.

  • The only published case is one website, one engine (ChatGPT) and four changes made together.
  • No study we reviewed explains why AI bots request addresses that do not exist.
  • No study links bot request volume for a page to how often it is cited.
  • Log evidence on llms.txt comes from industry sources, not controlled tests.
  • The 62.4% bot share is one network’s figure, quoted by a vendor.

Frequently asked questions

Can I see ChatGPT crawling my website in server logs?

Yes, OpenAI publishes separate names for its bots, such as GPTBot, OAI-SearchBot and ChatGPT-User, and each does a different job; 15.2% of top sites block GPTBot while only 7.4% block OAI-SearchBot.

Do AI bot 404 errors show what customers want?

They may hint at it, but the evidence is thin. One site built pages from the most frequent AI bot 404 patterns, as one of four changes, and could not measure that change on its own.

If an AI bot reads my page, will it cite it?

Not necessarily. On shopping questions, ChatGPT’s developer interface returned 37.38 pages per answer from its searches but cited 3.22.

How do I know whether AI traffic growth is from my changes?

Compare changed pages with similar pages you left alone. On one site, untouched pages grew 3.5 times, which would otherwise have been credited to the changes.

Sources

Free strategy call

Some questions are easier to answer about your own business.

Bring the one that matters most. On a free 30-minute call we’ll take a first look at it and send you a short written read afterward.

Ask about my businessHow we work