We Tested AI Brand Visibility With Real Users vs. the API. The Gap Was Surprising

We measure brand visibility for a living. So when a startup called BLESS Network offered to run our clients' prompts through a network of real, logged-in ChatGPT users instead of the API, we said yes.

We expected a small calibration difference — maybe 10 or 15 percentage points. A rounding error worth noting and moving on from.

We got a 30-point gap, a data quality crisis buried in Finnish server traffic, and a methodological lesson that every GEO measurement tool in this industry needs to wrestle with.

What does "real user" mean, and why does it matter?

Every GEO measurement tool on the market today — Profound, Peec.ai, Otterly.ai, Rankscale, and GEOforge — measures brand visibility the same way at its core: by sending prompts to the LLM API and analyzing the responses.

That approach has obvious advantages. It's fast, consistent, scalable, and cheap enough to run hundreds of prompt variants across multiple LLMs on a weekly cadence. It's also how the whole industry built itself.

The problem is what an API call isn't.

An API call has no user. No login. No account history. No memory of previous conversations. No personalization signals. No account plan tier. It's a completely stateless request to a model that, in a real-world session, would be responding to a logged-in person with a browsing history, a plan level (free or Pro), and potentially months of prior conversation context.

"ChatGPT's API is ChatGPT in a lab. Real users get something different. We suspected the gap existed. We just didn't know how large it was."

BLESS Network is a startup building what they describe as a distributed panel of real, logged-in LLM users — people running a Chrome extension that routes prompts through their active sessions across ChatGPT, Claude, and Perplexity. It's a fundamentally different measurement architecture than anything else in the GEO space.

We agreed to a trial: they would run the same 30 prompts we use for four client brands through their panel, and we'd compare mention rates.

What we tested

The trial covered four brands across two categories: two B2B software companies and two B2C consumer brands. For each brand, GEOforge had measured mention rates across 30 curated prompts using our standard SignalForge methodology — 50 ChatGPT API runs per prompt, with Wilson score confidence intervals.

BLESS ran those same 30 prompts through their network and returned CSV files with individual responses across ChatGPT, Claude, and Perplexity. Total response volume ranged from roughly 37,000 to 88,000 responses per brand across all three LLMs — a meaningfully large dataset.

The headline numbers: a significant gap

Here's the direct comparison for ChatGPT only — apples to apples, same prompts, same brand, different execution environment.

Target markets defined per brand based on actual operating geographies. "excl. FI" explained in the next section.

The raw gap is enormous. SignalForge reports a 65% mention rate for our established B2B client; BLESS reports 17%. The B2C coliving brand comes in at 75% on SignalForge and 28% on BLESS. For GEOforge itself — a small, niche brand — the gap goes from 16.5% to effectively zero.

Our first instinct was to blame geography. BLESS's network skews heavily toward Asia and Africa. Maybe LLMs just don't surface Western brands as often to users in those markets.

That turned out to be wrong. But something else in the data was worse.

Why does Finland account for half your dataset?

When we segmented the BLESS ChatGPT data by user country code, Finland appeared at the top of every brand's dataset — and not by a small margin.

Finnish users represented 38–68% of all ChatGPT responses in the trial, depending on the brand. Every brand, regardless of their target market or relevance to Nordic users, showed roughly 25,000 Finnish responses.

That uniform volume was the first red flag. The second was the mention rates:

Finnish users returned near-zero mention rates for every brand — including our B2C client operating housing communities across multiple European cities where Finnish users would be a plausible audience. The third flag was the account metadata: nearly all Finnish users had a blank userPlan field, while non-Finland users consistently showed "free" or "pro."

These weren't users. They were compute nodes — likely BLESS's own distributed server infrastructure running sessions in parallel. The organic human panel and the automated infrastructure traffic were mixed together in the same CSV without any labeling.

Does geography actually explain the remaining gap?

After removing Finland, we went looking for the geographic effect we'd originally expected to find. We segmented every remaining response by region — Asia, Africa, North America, Europe, Latin America — and compared mention rates within each region.

For our established B2B client on ChatGPT (excluding Finland), the rates were:

• Asia: 37.2%
• Africa: 34.9%
• North America: 34.1%
• Latin America: 25.4%
• Europe (ex. Finland): 35.4%

The gap between Asia and North America is 3 percentage points. For our European B2C client, Asia comes in at 47.1%, Africa at 44.6%, North America at 44.3%. The geographic spread is essentially flat.

We then filtered down to target-market users only for each brand. The rates barely moved from the global-minus-Finland average.

Geography explains roughly 10–15% of the total gap between API and real-user measurement. Not 50%. Not 80%. It's a real variable but not the dominant one.

What actually drives the gap?

After correcting for contaminated data and ruling out geography, a genuine 20–30 point gap persists between what the API reports and what real users see for established brands. Three factors appear to drive most of it.

Memory and user context

The API runs in a completely stateless environment. No account, no history, no context. Real ChatGPT users have personalized sessions. When a user with months of prior conversations about enterprise software asks about billing platforms, the model's response is influenced by that history. The API has no analogous signal. It appears to be more exploratory and willing to surface niche players — which inflates visibility scores for smaller brands, potentially significantly.

Account plan tier

This one surprised us. For our B2B client on ChatGPT, free-tier users showed a 41% mention rate versus 24% for Pro users. That's a 17-point split within the same platform, driven entirely by plan tier. Free users may be running different model versions, with different training optimizations or different response generation parameters. The API has no plan tier. It's running against a model endpoint that may not match what your target audience — who are disproportionately on paid plans — actually sees.

Brand size and model familiarity

The API-versus-real-user gap scales with brand size. For our established B2B client — a brand with years of training data — the corrected gap is roughly 30 points. For GEOforge, a young brand with limited web presence, the real-user mention rate is essentially zero regardless of correction. The API is more willing to surface niche brands in a stateless exploratory query. Real users, with real context, appear to pull responses toward well-established names more strongly.

What does this mean for GEO measurement?

The honest answer is that API-based measurement isn't wrong — it's just measuring something specific: how the model behaves in a clean-room, stateless, unauthenticated environment. That's a real and useful signal. It's consistent, repeatable, and comparable across time.

But it's not measuring what your customer sees when they ask ChatGPT which billing platform to use.

The multi-LLM data from this trial makes the point even sharper. BLESS returned results across ChatGPT, Claude, and Perplexity. The cross-platform variance is dramatic:

B2C Client B is mentioned in 45.7% of ChatGPT responses but only 2.1% of Claude responses. If you're only measuring ChatGPT — or only measuring the API — you're missing the full picture entirely.

This is what we're building toward with SignalForge's 75-observation architecture across five LLMs: statistically reliable, cross-platform measurement. The BLESS trial confirms that multi-platform coverage isn't a nice-to-have. For any brand where the audience is spread across multiple AI tools, single-platform API measurement produces a fundamentally incomplete picture.

What we're doing about it

We're not scrapping API-based measurement. The consistency, speed, and cost profile make it the right foundation for weekly operational tracking. What we are doing is treating the API signal and the real-user signal as two different things that need to be understood in relation to each other.

The practical path forward looks like this:

1. Use API measurement for trend tracking. Week-over-week and month-over-month movement in API-derived scores is a valid signal of whether your GEO work is improving your position — even if the absolute number overstates your real-world visibility.
2. Use real-user panels for calibration. Periodic benchmark runs against a clean real-user panel give you a correction factor — a sense of how far your API scores diverge from reality for your specific brand and category.
3. Measure across platforms, not just ChatGPT. The cross-platform variance in this dataset is too large to ignore. A brand scoring 45% on ChatGPT and 2% on Claude needs a different content strategy than one scoring uniformly across platforms.
4. Disclose the methodology. Every GEO measurement report should state clearly whether scores are API-derived or real-user-derived. The industry hasn't standardized on this disclosure yet. It needs to.

We'll continue the conversation with BLESS as their panel matures and the data quality issues get resolved. The concept is sound. Real-user-derived measurement is where the industry needs to go. We just need the data to be clean enough to trust.