GEOforge Research · AI Crawlers
More AI bots on a page does not mean more AI citations. Crawling is the ticket, not the prize.
Every GEO dashboard now has a crawler chart, and every marketer looking at one asks the same question: if more AI bots visit my pages, will I get more AI citations? We joined the two datasets to find out. Over 11 weeks (14 August to 30 October 2026) that meant more than 1 million AI bot requests, 945,000 of them to 45,400 content pages on 12 websites, matched against 58,300 citations of those pages in 284,600 tracked answers from ChatGPT, Google AI Overviews and Google AI Mode.
Yes, AI crawl activity correlates with AI citations. No, turning it up won’t get you cited.
The dataset covers 12 websites running GEOforge: B2B SaaS, cybersecurity, health diagnostics, a supplements retailer in two countries, a private surgical practice, an education consultancy, a recruitment firm and three agencies or agency advisers, including our own two sites.
Crawl activity. GEOforge logs AI crawler requests through a WordPress plugin, a Cloudflare integration or a server-side snippet. We grouped 20 AI user agents into three families, because they do very different jobs.
| Bot family | Examples | Share of requests |
|---|---|---|
| Training crawlers | GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot | 74% |
| AI search indexers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot, DuckAssistBot | 22% |
| User-triggered fetchers | ChatGPT-User, Claude-User, Perplexity-User | 4% |
We dropped any request whose IP address failed verification against the operator’s published ranges, plus feeds, assets, tag and author archives. URLs were normalized (protocol, www, query strings and trailing slashes stripped) so each page counts once.
Citations. GEOforge runs each brand’s tracked prompts many times per cycle in ChatGPT, Google AI Overviews and Google AI Mode, and records every URL each answer cites. A citation here is one answer citing one of the brand’s own pages.
The comparison. Sites differ wildly. One supplements store took 449,000 bot requests in the period; one agency site took 1,200. So every comparison is made within a site, then combined. The main statistic is simple: pick a cited page and an uncited page from the same site at random. How often does the cited one have more crawl activity? 50% is a coin flip. 100% is a perfect predictor.
Yes, on every site without exception. On the coin-flip test, a cited page out-crawled an uncited page from the same site 76% of the time (median across sites). The range ran from 58% on the Danish supplements store to 86% on the surgical practice, which had 68 cited pages, and higher still on smaller sites.
The most-crawled 1% of a site’s pages were cited 34× as often as its bottom half
Share of pages cited at least once, by the page’s AI crawl rank within its own site. Hover a bar for counts.
12 sites, 45,390 crawled content pages, 14 August to 30 October 2026. Tiers are computed within each site, so no single site dominates. 41 cited pages with no logged AI bot visit are excluded from the tiers.
Citations concentrate just as hard. The 10% most-crawled pages on each site collected 59% of all citations. The least-crawled half collected under 3%. If you stopped reading here, you’d conclude that crawl activity drives citations. Keep reading.
No. Even among the most-crawled 1% of pages, 92% were never cited once. Crawl volume separates the contenders from the also-rans. It doesn’t make anyone win.
The sites with the most bot traffic make the point. The two supplements stores absorbed 672,000 requests between them, 71% of the content-page total, mostly training crawlers working through product catalogs. They showed the weakest link between crawl and citation of any site in the study (58% and 60% on the coin-flip test). Meta’s training crawler alone sent 41% of every AI bot request we logged. None of the three answer engines we track runs on it.
Among pages that did get cited, more crawling only loosely meant more citations. The median rank correlation between crawl volume and citation count was 0.31, and it was negative on two sites.
Mostly not; it measures how prominent a page already is. Google AI Overviews and AI Mode are built on Google’s own index, gathered by Googlebot, which isn’t in these logs at all. If AI bot visits caused citations, they should predict ChatGPT citations well and Google citations poorly. They didn’t.
| Bot activity on the page | ChatGPT citations | Google AI Overviews & AI Mode citations |
|---|---|---|
| All AI bots except user-triggered fetchers | 68% | 75% |
| OpenAI’s crawlers (GPTBot, OAI-SearchBot) | 74% | 72% |
| ChatGPT-User fetches | 78% | 79% |
Scores are the coin-flip test, averaged across sites and weighted by each site’s cited pages. 50% means no signal.
OpenAI’s own crawlers predicted Google citations almost exactly as well as ChatGPT citations, and Google doesn’t consult GPTBot. The simplest explanation is a shared cause. Bots and answer engines are drawn to the same pages: the ones with strong internal links, backlinks, a clear topic and existing demand.
Crawl activity is a thermometer, not a thermostat. It tells you where the heat is. It doesn’t create any.
The bot families still differ. AI search indexers (74%) and user-triggered fetchers (77%) beat training crawlers (67%), and search indexers beat training crawlers on 10 of the 12 sites. So if you watch one line on your crawler chart, watch search and live-answer fetches, not training volume. User-triggered fetches are the closest thing to a free, prompt-independent demand signal: they track citations even in Google’s AI features, which never send them.
Yes, and sharply. We took the 115 pages whose first tracked citation landed between 11 September and 16 October and lined up their crawl history around that date. 105 of them (91%) had an AI bot visit in the four weeks before their first citation.
AI search bots nearly tripled their visits to a page before its first citation
Weekly AI search-indexer visits, indexed to 100 four weeks before the first observed citation. Hover a point for raw counts.
115 pages first cited 11 September to 16 October 2026. “Rest of the site” is all other pages on the same 12 sites over the same weeks.
AI search-indexer visits to these pages were up 55% two weeks out, while the same bots’ visits to the rest of each site fell. Training crawlers doubled on these pages too, from 105 to 222 visits a week, against a 24% rise site-wide.
Almost never. The cleanest evidence comes from new pages, where we know exactly when each one went live. We followed 129 pages published through GEOforge between 14 August and 9 October on eight sites with crawl tracking in place.
| Milestone after publishing | Result |
|---|---|
| First AI bot visit | 6 hours (median) |
| First AI search indexer visit | 24 hours (median) |
| Pages cited by 30 October | 52% |
| First citation, for pages that were cited | 5.8 days (median) |
126 of the 129 pages had an AI bot visit within two weeks. Once a page cleared that bar, early crawl volume said almost nothing about whether it would be cited: 58% of pages with fewer than five bot visits in their first two weeks were cited, against 50% with 5 to 14 visits and 51% with 15 or more. The heavily crawled pages did collect more citations on average (139 per page against 70), which fits the thermometer reading.
Now the counterexample. One site published 130 pages on a dedicated content subdomain. 83 of them were never fetched by a single AI bot during the study, and not one of those 83 was cited. The subdomain had little visibility overall, so treat this as a demonstration rather than a controlled test, but the direction is unambiguous.
The gate isn’t absolute. 41 of the 474 cited pages in the study (9%) had no logged AI bot visit in the window. Most were cited by ChatGPT, which also draws on third-party search indexes, and the rest by Google, whose crawler isn’t in these logs. Some engines walk through a gate you can’t see.
AI crawl activity does two jobs, and neither of them is “lever”.
It’s a gate. A page that no AI system has fetched has almost no chance of being cited. On healthy sites the gate opens within hours, so most teams will never notice it. On a new domain, a fresh subdomain, behind an overzealous firewall rule or under a robots.txt line someone added in 2023, it can stay shut for good.
It’s a thermometer. Once the gate is open, crawl volume rises and falls with the same things that earn citations: relevance to real questions, internal and external links, and demand.
What it is not is a thermostat. Nothing in this data suggests that attracting more bot visits to a page that already gets crawled will move its citations. A page that answers the question gets crawled and cited. A page that doesn’t gets crawled and ignored. In this study, nearly 45,000 crawled pages were never cited once.
Data. GEOforge’s production database, 14 August to 30 October 2026: AI crawler logs and citation records for 12 sites, 45,390 crawled content pages, 474 cited pages and 284,600 tracked answers. Two sites came online during the window, in late August and mid October.
Method. Each comparison is made within a site and then averaged, weighted by cited pages. The coin-flip score is the probability that a randomly chosen cited page has more crawl activity than a randomly chosen uncited page on the same site (the area under the ROC curve).
Limitations. This is a correlational study of 12 sites, and the two supplements stores account for two thirds of all pages. Citations are measured only on each brand’s tracked prompts, re-run weekly, so a “first citation” date can trail the real one by up to a week. Googlebot is out of scope. Most requests arrive through integrations where the IP can’t be verified, so a small amount of spoofed traffic likely remains. User-triggered fetchers can fire during our own tracking runs, so their link to citations is partly a consequence of citation, not a cause. No site is named.
GEOforge shows AI crawler activity and AI citations for the same pages, across ChatGPT, Google AI Overviews and Google AI Mode, so you can find the pages bots never reach and the ones they reach that never get cited. 14-day free trial, no sales call.
Start a free trial →Sources & method. GEOforge production crawl and citation database, 14 August to 30 October 2026, 12 sites. See also our analysis of how Google AI Overviews and AI Mode cite different websites and our measurement methodology.