AI Crawlers Don’t Get You Cited. They Just Show Up First.

Paris Childress
November 18, 2026

GEOforge Research · AI Crawlers

More AI bots on a page does not mean more AI citations. Crawling is the ticket, not the prize.

34×
How much more often a site’s most-crawled 1% of pages were cited than its least-crawled half
92%
Share of even the most heavily crawled pages that were never cited once
0 of 83
New pages cited that no AI bot ever fetched
6 hours
Median time from publishing to first AI bot visit on established sites

Every GEO dashboard now has a crawler chart, and every marketer looking at one asks the same question: if more AI bots visit my pages, will I get more AI citations? We joined the two datasets to find out. Over 11 weeks (14 August to 30 October 2026) that meant more than 1 million AI bot requests, 945,000 of them to 45,400 content pages on 12 websites, matched against 58,300 citations of those pages in 284,600 tracked answers from ChatGPT, Google AI Overviews and Google AI Mode.

What is the short answer?

Yes, AI crawl activity correlates with AI citations. No, turning it up won’t get you cited.

  • The correlation is real and universal. On all 12 sites, cited pages were crawled more than uncited ones. A site’s top 1% most-crawled pages were cited 34 times as often as its bottom half.
  • It’s a lousy dial. 92% of even the most heavily crawled pages were never cited once.
  • It’s mostly a thermometer. AI bot activity predicted citations in Google’s AI features slightly better than in ChatGPT, even though Google doesn’t use any of those bots. Crawl volume mostly reflects how prominent a page already is.
  • The one hard rule is binary. 83 new pages that no AI bot ever fetched earned zero citations. On established sites, new pages typically saw an AI bot within six hours, and the half that got cited were usually cited within six days.

What did we measure?

The dataset covers 12 websites running GEOforge: B2B SaaS, cybersecurity, health diagnostics, a supplements retailer in two countries, a private surgical practice, an education consultancy, a recruitment firm and three agencies or agency advisers, including our own two sites.

Crawl activity. GEOforge logs AI crawler requests through a WordPress plugin, a Cloudflare integration or a server-side snippet. We grouped 20 AI user agents into three families, because they do very different jobs.

AI bot families in the dataset
Bot familyExamplesShare of requests
Training crawlersGPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot74%
AI search indexersOAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot, DuckAssistBot22%
User-triggered fetchersChatGPT-User, Claude-User, Perplexity-User4%

We dropped any request whose IP address failed verification against the operator’s published ranges, plus feeds, assets, tag and author archives. URLs were normalized (protocol, www, query strings and trailing slashes stripped) so each page counts once.

Citations. GEOforge runs each brand’s tracked prompts many times per cycle in ChatGPT, Google AI Overviews and Google AI Mode, and records every URL each answer cites. A citation here is one answer citing one of the brand’s own pages.

The comparison. Sites differ wildly. One supplements store took 449,000 bot requests in the period; one agency site took 1,200. So every comparison is made within a site, then combined. The main statistic is simple: pick a cited page and an uncited page from the same site at random. How often does the cited one have more crawl activity? 50% is a coin flip. 100% is a perfect predictor.

Do cited pages get crawled more?

Yes, on every site without exception. On the coin-flip test, a cited page out-crawled an uncited page from the same site 76% of the time (median across sites). The range ran from 58% on the Danish supplements store to 86% on the surgical practice, which had 68 cited pages, and higher still on smaller sites.

The most-crawled 1% of a site’s pages were cited 34× as often as its bottom half

Share of pages cited at least once, by the page’s AI crawl rank within its own site. Hover a bar for counts.

Share of pages cited at least once, by AI crawl rank within their own site Pages in the least-crawled half of each site were cited 0.24 percent of the time. Pages in the top 1 percent by AI crawl volume were cited 8.1 percent of the time, 34 times as often. Bottom 50% most-crawled pages: 55 of 23,242 cited (0.24%) 0.24% Bottom 50% 3% of citations 25–50% most-crawled pages: 113 of 11,257 cited (1.0%) 1.0% 25–50% 15% of citations 10–25% most-crawled pages: 127 of 6,849 cited (1.9%) 1.9% 10–25% 23% of citations 5–10% most-crawled pages: 43 of 1,873 cited (2.3%) 2.3% 5–10% 13% of citations 1–5% most-crawled pages: 63 of 1,775 cited (3.5%) 3.5% 1–5% 30% of citations Top 1% most-crawled pages: 32 of 394 cited (8.1%) 8.1% Top 1% 16% of citations Least crawled → most crawled. Pages ranked by AI bot requests within their own site.

12 sites, 45,390 crawled content pages, 14 August to 30 October 2026. Tiers are computed within each site, so no single site dominates. 41 cited pages with no logged AI bot visit are excluded from the tiers.

Citations concentrate just as hard. The 10% most-crawled pages on each site collected 59% of all citations. The least-crawled half collected under 3%. If you stopped reading here, you’d conclude that crawl activity drives citations. Keep reading.

Will more crawling get a page cited?

No. Even among the most-crawled 1% of pages, 92% were never cited once. Crawl volume separates the contenders from the also-rans. It doesn’t make anyone win.

The sites with the most bot traffic make the point. The two supplements stores absorbed 672,000 requests between them, 71% of the content-page total, mostly training crawlers working through product catalogs. They showed the weakest link between crawl and citation of any site in the study (58% and 60% on the coin-flip test). Meta’s training crawler alone sent 41% of every AI bot request we logged. None of the three answer engines we track runs on it.

Among pages that did get cited, more crawling only loosely meant more citations. The median rank correlation between crawl volume and citation count was 0.31, and it was negative on two sites.

Does crawl activity cause citations?

Mostly not; it measures how prominent a page already is. Google AI Overviews and AI Mode are built on Google’s own index, gathered by Googlebot, which isn’t in these logs at all. If AI bot visits caused citations, they should predict ChatGPT citations well and Google citations poorly. They didn’t.

How well bot activity predicts citations, by engine
Bot activity on the pageChatGPT citationsGoogle AI Overviews & AI Mode citations
All AI bots except user-triggered fetchers68%75%
OpenAI’s crawlers (GPTBot, OAI-SearchBot)74%72%
ChatGPT-User fetches78%79%

Scores are the coin-flip test, averaged across sites and weighted by each site’s cited pages. 50% means no signal.

OpenAI’s own crawlers predicted Google citations almost exactly as well as ChatGPT citations, and Google doesn’t consult GPTBot. The simplest explanation is a shared cause. Bots and answer engines are drawn to the same pages: the ones with strong internal links, backlinks, a clear topic and existing demand.

Crawl activity is a thermometer, not a thermostat. It tells you where the heat is. It doesn’t create any.

The bot families still differ. AI search indexers (74%) and user-triggered fetchers (77%) beat training crawlers (67%), and search indexers beat training crawlers on 10 of the 12 sites. So if you watch one line on your crawler chart, watch search and live-answer fetches, not training volume. User-triggered fetches are the closest thing to a free, prompt-independent demand signal: they track citations even in Google’s AI features, which never send them.

Does crawling pick up before a page is first cited?

Yes, and sharply. We took the 115 pages whose first tracked citation landed between 11 September and 16 October and lined up their crawl history around that date. 105 of them (91%) had an AI bot visit in the four weeks before their first citation.

AI search bots nearly tripled their visits to a page before its first citation

Weekly AI search-indexer visits, indexed to 100 four weeks before the first observed citation. Hover a point for raw counts.

Weekly AI search-indexer visits around a page's first observed citation Indexed to 100 four weeks before first observed citation. On the 115 pages about to be cited, AI search-indexer visits rose to 155 two weeks before and 294 in the final week, while visits to the rest of the same sites fell to 55. 0 100 200 300 First observed citation Rest of the site, week -4: 170,108 search-indexer visits (index 100) Rest of the site, week -3: 176,657 search-indexer visits (index 104) Rest of the site, week -2: 133,007 search-indexer visits (index 78) Rest of the site, week -1: 93,618 search-indexer visits (index 55) Rest of the site, week +0: 91,234 search-indexer visits (index 54) Rest of the site, week +1: 77,755 search-indexer visits (index 46) 46 Rest of the site Pages about to be cited, week -4: 62 search-indexer visits (index 100) Pages about to be cited, week -3: 76 search-indexer visits (index 123) Pages about to be cited, week -2: 96 search-indexer visits (index 155) Pages about to be cited, week -1: 182 search-indexer visits (index 294) Pages about to be cited, week +0: 151 search-indexer visits (index 244) Pages about to be cited, week +1: 142 search-indexer visits (index 229) 229 Pages about to be cited 294 -4 -3 -2 -1 0 +1 Weeks relative to first observed citation (citations are tracked weekly). Index: week −4 = 100.

115 pages first cited 11 September to 16 October 2026. “Rest of the site” is all other pages on the same 12 sites over the same weeks.

AI search-indexer visits to these pages were up 55% two weeks out, while the same bots’ visits to the rest of each site fell. Training crawlers doubled on these pages too, from 105 to 222 visits a week, against a 24% rise site-wide.

Read the ramp carefully. Most of these pages were new: 70 of them were first seen by any AI bot less than five weeks before their first citation, and new pages attract a burst of bot discovery whether or not they’ll ever be cited. Because citations are tracked weekly, the steepest jump in the final week also overlaps the moment of first citation itself. Crawling comes before citation. That doesn’t make it the cause.

Can a page be cited without being crawled?

Almost never. The cleanest evidence comes from new pages, where we know exactly when each one went live. We followed 129 pages published through GEOforge between 14 August and 9 October on eight sites with crawl tracking in place.

New pages on established sites
Milestone after publishingResult
First AI bot visit6 hours (median)
First AI search indexer visit24 hours (median)
Pages cited by 30 October52%
First citation, for pages that were cited5.8 days (median)

126 of the 129 pages had an AI bot visit within two weeks. Once a page cleared that bar, early crawl volume said almost nothing about whether it would be cited: 58% of pages with fewer than five bot visits in their first two weeks were cited, against 50% with 5 to 14 visits and 51% with 15 or more. The heavily crawled pages did collect more citations on average (139 per page against 70), which fits the thermometer reading.

Now the counterexample. One site published 130 pages on a dedicated content subdomain. 83 of them were never fetched by a single AI bot during the study, and not one of those 83 was cited. The subdomain had little visibility overall, so treat this as a demonstration rather than a controlled test, but the direction is unambiguous.

The gate isn’t absolute. 41 of the 474 cited pages in the study (9%) had no logged AI bot visit in the window. Most were cited by ChatGPT, which also draws on third-party search indexes, and the rest by Google, whose crawler isn’t in these logs. Some engines walk through a gate you can’t see.

What does this mean for GEO?

AI crawl activity does two jobs, and neither of them is “lever”.

It’s a gate. A page that no AI system has fetched has almost no chance of being cited. On healthy sites the gate opens within hours, so most teams will never notice it. On a new domain, a fresh subdomain, behind an overzealous firewall rule or under a robots.txt line someone added in 2023, it can stay shut for good.

It’s a thermometer. Once the gate is open, crawl volume rises and falls with the same things that earn citations: relevance to real questions, internal and external links, and demand.

What it is not is a thermostat. Nothing in this data suggests that attracting more bot visits to a page that already gets crawled will move its citations. A page that answers the question gets crawled and cited. A page that doesn’t gets crawled and ignored. In this study, nearly 45,000 crawled pages were never cited once.

What should you do about it?

  • Open the gate and check it stays open. Allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and Googlebot in robots.txt. Then check your CDN and firewall bot rules, which can block AI crawlers regardless of what robots.txt says. Any important page with zero AI search-indexer visits in 30 days is a problem worth fixing today.
  • Get new pages discovered in hours, not weeks. Update your XML sitemap on publish, link every new page from a page that’s already crawled heavily, and ping Bing through IndexNow. If your new pages take days to see their first AI bot, start there.
  • Stop celebrating training-crawler spikes. Three quarters of AI bot traffic is training crawlers, and they were the weakest predictor of citations. A surge from Meta-ExternalAgent or Bytespider is a bandwidth bill, not a visibility win.
  • Use search and live-answer fetches as a demand signal. Pages that AI search indexers and ChatGPT-User keep coming back to are the pages answer engines lean on. Keep them current, expand them, and link from them to the pages you want cited next.
  • Put your effort into what earns the citation. Once a page is crawled, crawl volume stops being your constraint. Relevance to the prompts your buyers actually ask, original information and a clear answer decide whether the visit turns into a citation.

How did we measure this?

Data. GEOforge’s production database, 14 August to 30 October 2026: AI crawler logs and citation records for 12 sites, 45,390 crawled content pages, 474 cited pages and 284,600 tracked answers. Two sites came online during the window, in late August and mid October.

Method. Each comparison is made within a site and then averaged, weighted by cited pages. The coin-flip score is the probability that a randomly chosen cited page has more crawl activity than a randomly chosen uncited page on the same site (the area under the ROC curve).

Limitations. This is a correlational study of 12 sites, and the two supplements stores account for two thirds of all pages. Citations are measured only on each brand’s tracked prompts, re-run weekly, so a “first citation” date can trail the real one by up to a week. Googlebot is out of scope. Most requests arrive through integrations where the IP can’t be verified, so a small amount of spoofed traffic likely remains. User-triggered fetchers can fire during our own tracking runs, so their link to citations is partly a consequence of citation, not a cause. No site is named.

Your crawler chart shows who visited. Track who cited.

GEOforge shows AI crawler activity and AI citations for the same pages, across ChatGPT, Google AI Overviews and Google AI Mode, so you can find the pages bots never reach and the ones they reach that never get cited. 14-day free trial, no sales call.

Start a free trial →

Sources & method. GEOforge production crawl and citation database, 14 August to 30 October 2026, 12 sites. See also our analysis of how Google AI Overviews and AI Mode cite different websites and our measurement methodology.

Paris Childress
CEO

Paris Childress is the CEO of Hop AI and creator of GEOforge, a platform that helps B2B brands get cited and recommended by AI assistants like ChatGPT, Perplexity, and Gemini. A former Google Country Manager and agency veteran with 20+ years in digital marketing, Paris is focused on helping brands win in the era of AI search.