AI Visibility Tool Accuracy: Why Trackers Report Different Results

Quick answer: AI visibility tool accuracy depends on the engines, prompts, repeated runs and settings behind each score. Trackers report different results because each one samples different conditions. Compare trends inside one consistent setup. Ask vendors how they measure uncertainty, and keep brand mentions separate from citations to specific pages.

Across 2,961 prompt runs in SparkToro and Gumshoe.ai’s research, the chance of ChatGPT or Google’s AI returning the same brand list twice was under 1 in 100. In a separate test of 35 prompts, Paprik AI found 135 unique brands through a bare API without web search and 845 in the ChatGPT interface with search on.

That gap shows up in real meetings. A CMO checks a prompt in a personal ChatGPT account and sees brands the tracking dashboard never reported. A visibility drop gets blamed on content when the model behind the engine changed. A team rewrites a page that never lost a citation. When two answer engine optimization (AEO) or generative engine optimization (GEO) dashboards report conflicting AI share of voice (AI SoV), they are sampling different retrieval conditions. They are not measuring the same thing twice.

Every AI visibility tracker samples its own mix of engines, prompts, repeated runs and settings, and each score estimates brand presence under those conditions only. Before you call a change progress or a setback, check the conditions behind the answers and the math behind the percentage.

Seven reasons AI visibility scores differ

Answer variance changes results between runs

The same prompt does not return a fixed brand list. LLMs generate answers stochastically, picking each word from a probability distribution, so repeated runs can name different brands in a different order. A visibility score needs repeated observations. One answer is a snapshot, not a permanent position.

SparkToro and Gumshoe.ai had 600 volunteers run 12 prompts through ChatGPT, Claude and Google’s AI Overviews or AI Mode, 2,961 times in total. For ChatGPT and Google’s AI, the chance of getting the same brand list twice was under 1 in 100. Claude repeated itself only slightly more often.

SparkToro’s Rand Fishkin argues that a “ranking position in AI” is unreliable because the order is largely random, which is why traditional LLM rank tracking misleads. The study backs visibility percentage, measured across many prompts run many times, instead. The industry also calls that repeated-run measure Share of Model (SoM). Keep it in mind when a dashboard presents an AI recommendation as if it were a search ranking.

When a score moves, ask whether the move reflects repeated observations or one changed answer. Keep the raw answers behind every score so your team sees what changed.

The API and the consumer app measure different experiences

A raw model API response and a ChatGPT session are different measurement environments. A tracker should tell you which one its results represent.

Paprik AI ran the same 35 prompts for 5 brands over 10 days. The bare API, with no web search, surfaced 135 unique brands. The ChatGPT interface, with web search on, surfaced 845.

Paprik calls the gaps “ghost competitors” (rivals the API data misses) and “zombie competitors” (brands the API data overstates). The takeaway is simple. The two environments produced very different brand sets.

Ask vendors for the exact interface, its settings and the experience it represents. An API-based score is evidence about that setup. It is not a copy of every customer’s ChatGPT session.

Web search and grounding change the evidence

Search being available and search actually running are separate questions. Turning a tool on does not mean every answer used it.

OpenAI documents web search as an optional tool. The model decides whether to search based on the prompt. Search-backed answers include inline citations by default.

Google’s grounding documentation has developers add Google Search to the request explicitly. Once it is on, the model handles searching, processing and citing.

Grounding is a form of retrieval-augmented generation (RAG): the model pulls live sources into the answer instead of relying only on training data. A vendor should separate tool availability from tool use. Ask whether each stored answer records a search event and its citations.

Also separate an unlinked brand mention from a grounded source citation to your specific URL. A mention builds awareness, and in a zero-click AI answer it may be the only exposure your brand gets. A clickable citation is what sends referral traffic and leads. A dashboard that blends the two hides which answers can actually send visitors to your site.

Location, language and login state change the context

A neutral tracking session and a customer’s personalized session serve different purposes. Location settings need explicit documentation too.

Anthropic’s web search documentation says developers add the tool to the request before Claude has it. Claude decides when to search based on the prompt, and the user_location parameter localizes the results.

Otterly says ChatGPT and Perplexity personalize answers using stored information such as search history and location. Otterly says it aims to provide neutral, objective monitoring instead.

This is where reporting friction starts. An executive who checks a prompt in a personal ChatGPT or Perplexity account sees answers shaped by that account’s history and location, and the neutral dashboard will not match. Agree in advance which view the report represents, and show leadership the stored answer behind each score.

Ask which city, region, country, language and login state each report represents. For local visibility, ask for the location actually sent with the query.

Hold those settings constant when you compare periods. Our guide to tracking your brand in ChatGPT and Perplexity covers how to plan that monitoring.

Prompt-set construction defines what visibility means

A score only describes performance on the questions in its calculation. Change the questions and you change the measurement, even when your brand and website stay the same.

Ahrefs says all AI visibility tools, its own included, use synthetic prompts. The reason: no company has actual search volume data for ChatGPT or other AI platforms yet.

Synthetic prompts are constructed questions. They are only as useful as their match to your buyers, products, locations and purchase decisions.

Ask for sample prompts across informational questions, category recommendations and purchase comparisons. Ask whether the prompts name your brand. Branded questions and open category questions test different things.

Ask how prompts enter and leave the tracked set, too. A score that rises after hard questions are removed is not comparable with the earlier score.

A documented measurement panel beats an unexplained pile of questions. Our overview of AEO analysis tools and AI visibility shows how to evaluate these reports.

Sample size sets the uncertainty around a score

A precise-looking percentage is not the same as an accurate one. The number of observations behind it sets how much uncertainty surrounds the estimate.

Gumshoe.ai puts the sample needed for one prompt on one model at 97 queries for a margin of error of plus or minus 10 percentage points, 385 for 5 points and 9,604 for 1 point. Those figures assume worst-case variance.

They apply to repeated runs of a single prompt. They do not give you the margin of error for a dashboard that blends unrelated prompts, engines and markets.

Precision costs money. Cutting the margin from 10 points to 5 takes roughly four times the runs. At 385 runs per prompt, a 50-prompt panel on three engines needs 57,750 queries per reporting period. Decide which prompts need tight intervals, such as your core purchase comparisons, and accept wider margins on the rest.

A confidence interval states the uncertainty around an estimated proportion. NIST’s e-Handbook of Statistical Methods explains confidence intervals for proportions, including the Wilson method. That is the statistical basis for reading sampled visibility percentages.

Ask for the denominator, the confidence level and the interval method. Ask whether the calculation covers single prompts, one engine or the full dashboard.

Not every small movement is a business event. Confirm the evidence supports the story before you report it.

Model updates break otherwise consistent tracking

An unchanged website does not mean an unchanged measurement system. The model behind a familiar product name changes too, so it needs version tracking.

OpenAI’s deprecations page records model shutdowns, each with a shutdown date and a recommended replacement. Once a model is retired, any tracker built on it has to move to a new one.

Ask vendors whether they record the underlying model version and flag changes. Log those events next to your content publish dates when you investigate a sudden swing.

A model change is one more explanation for a visibility shift. Your latest edit is not automatically the cause.

Questions to ask before you buy a tracker

Judge AI visibility tool accuracy by the measurement method behind it. Ask each vendor to walk through a sample report against this checklist.

• Interface: Does tracking query a raw model API or the consumer app?
• Search: Is web search or grounding enabled, and does the report confirm it ran?
• Context: How are location, language, login state and personalization handled?
• Repetition: How many times does each prompt run per reporting period?
• Uncertainty: What margin of error or confidence interval comes with the result?
• Prompt selection: Are prompts synthetic, how were they built and how are they revised?
• Definitions: Does visibility mean a brand mention, a recommendation, a domain citation or an exact URL citation?
• Change records: Are model updates and measurement changes documented?

A good answer ties the score back to the underlying answers. A bare aggregate percentage leaves too much of the measurement undefined.

When two tools disagree, reporting errors compound

1. Different definitions become a false comparison. Check the metric, prompt set, interface, search settings and reporting window before you discuss the gap. Never compare absolute scores across tools as if they share a scale. Each has its own denominator and conditions, even when both dashboards call it visibility.

2. A sampling change becomes a performance story. Read trends inside one tool under consistent conditions. Before you credit content, check whether prompts, model versions, location settings or run counts changed. Keep the earlier configuration on file, and mark breaks instead of presenting them as uninterrupted progress.

3. A performance story becomes an unnecessary rewrite. Inspect the underlying answers and cited URLs before you change published content. Separate repeated citation patterns from a one-off recommendation. Act on consistent observations, and report unresolved differences honestly instead of averaging incompatible scores into a new number.

Use these questions when you read a comparison such as Profound vs Peec AI. Compare measurement definitions first, dashboard totals second.

At AEO Agent X, stage 18 of our 19-stage workflow tracks positions, clicks, indexation, leads and AI citations per URL. That ties citation reporting to the pages your team actually manages.

Lock one tracker setup before your next stakeholder report

Send the eight checklist questions to every vendor on your shortlist this week. Drop any tracker that cannot show its prompt set, interface, run counts and confidence interval on a sample report. Then fix one configuration, write it down and share it with leadership, so the next spot-check in a personal account starts a conversation about method instead of a dispute about numbers.

Want AI citations tracked per URL alongside positions, clicks and leads? Request access to AEO Agent X. We reply within one business day.

Frequently Asked Questions (FAQs)

Why do AI visibility tools show different numbers for the same brand?

Each tool samples different engines, prompts, repeated runs and settings. They also use different metric definitions and reporting windows. Compare trends inside one consistent setup, not absolute scores across tools.

Is the ChatGPT API the same as the ChatGPT app for tracking mentions?

No. The API and the consumer app are different measurement environments. Paprik AI found 135 unique brands through the bare API without search and 845 in the ChatGPT interface with search on. Ask which environment a tracker measures and which settings it uses.

What is the difference between an AI brand mention and an AI source citation?

A brand mention names your company in the answer text, often without a link. A source citation credits a specific URL as evidence for the answer. Mentions build awareness, and citations are what send referral traffic and leads. AEO Agent X tracks AI citations per URL, so reporting maps to the pages your team manages.

How many times should a prompt run for accurate AI share of voice and GEO reporting?

The run count follows the precision you need. Gumshoe.ai puts it at 97 queries for plus or minus 10 percentage points and 385 for 5 points, for one prompt on one model. Ask how the vendor calculates uncertainty for its broader score.

Are AI visibility tracking prompts real customer queries?

No. Ahrefs says all AI visibility tools use synthetic prompts, because no company has actual search volume data for AI platforms yet. Treat those prompts as a constructed measurement panel. Ask how the panel represents your buyers and whether it changes over time.

Why does an AI citation score change when my website stays unchanged?

Answer variation, model updates, prompt revisions and search settings all move scores without any site change. Location and personalization change the measured context too. Inspect the underlying answers and measurement records before you credit or blame your content.