AI Visibility Metrics: Why the Number Moves, and How to Report It Honestly

Put the same buyer question to the same assistant twice and the brand list can come back different. Nothing is broken. That instability is the thing your AI visibility metrics have to be built around, and almost every dashboard in this category hides it behind a single tidy number.

This is for the person who owns the monthly report: you track brand mentions across ChatGPT, Perplexity, Gemini or Claude, leadership asks whether the number went up, and you cannot honestly say. What follows is the mechanism that makes answers move, the formulas the trackers use underneath their score, and the sample size at which a rate stops being an anecdote.

Three sources of variation, stacked on top of each other

Answers move for three separate reasons, and they compound.

1. Generation is prediction, not lookup. Google's own guidance on generative output states that "generative models don't retrieve facts, but predict a likely sequence of words based on their training data". A different draw from the same distribution is a different sentence, and sometimes a different brand.
2. Retrieval is decided at answer time. An assistant that searches the live web picks its pages in the moment. Wording, time of day and location all move which pages it finds, and therefore which companies end up in the summary.
3. The product underneath you keeps shipping. Models get swapped and response formats get rebuilt. The arrival of ChatGPT's Intelligent UI in October 2026 changed the shape of a reply outright, as iClarified reported at the time. A measurement series that spans a change like that is two series, not one.

The practical consequence: a single answer is one sample from a distribution you cannot see directly. You learn the distribution by sampling it repeatedly, which means a run count is part of your metric, not a footnote to it.

What is inside a tracker's headline score

Most tools present one composite figure. It is worth knowing what they multiply to get it. Keyword.com publishes its working: detection rate is the count of runs where the brand was found divided by total runs, rank score is 1 divided by (1 plus 0.1 times average rank minus one), and the visibility score is those two multiplied and scaled to a hundred. A top-three rate counts the runs where the brand placed first, second or third against all runs.

Two things follow. First, a composite score mixes presence with placement, so a score can slip because you appeared less often or because you appeared lower, and the number itself will not say which. Second, every term in it is a ratio over runs. Change the run count and you change the precision of the score without changing anything about your brand. That is why two vendors reporting on the same brand can disagree, and why the terms behind these metrics are worth pinning down before you compare tools.

Surfer makes the same warning from the buying side in its guide to choosing a tracker: a visibility figure reflects point-in-time performance, so the trend line matters more than the reading.

Walk one prompt through ten runs

This sequence is illustrative, not measured. A two-person content team asks an assistant which tool suits a small approval workflow.

• First run. Three products named, none of them theirs. The reasonable conclusion: we are invisible, and somebody should rewrite the product page this week.
• Second run, ninety seconds later. Four products named, theirs among them. The reasonable conclusion: we are fine, cancel the rewrite.
• Runs three through ten. They land in four of the remaining eight.

The defensible statement is "present in five of ten runs on one prompt, small sample". The first two runs each supported a confident and opposite decision, and both decisions would have been made on noise. This is the mechanism by which a measurement error turns into a rewritten page that did not need rewriting.

The vendor data on how often brands drop out

Variation is not only your problem, and there is now a number attached to it. AirOps, which sells in this category, published figures in January 2026 from its own State of AI Search research: only 30% of brands stay visible from one answer to the next, and 20% hold across five consecutive runs. Treat those as a vendor's own data rather than independent measurement, because the method is not published in full.

Two further claims from the same report are more useful for planning than for reporting. AirOps puts roughly 85% of brand mentions on third-party pages rather than the brand's own domain, and finds brands earning both citations and mentions about 40% more likely to resurface across runs than brands earning citations alone. If that holds, the lever is not only your own pages. It is whether the pages that already rank for your category say your name, which is a different workstream from fixing what an assistant can read on your site.

Sample size, and how wide the band really is

Here is arithmetic you can reproduce, not a vendor figure. Detection rate is a proportion, so its uncertainty behaves like any proportion's. At a true rate of one in two, ten runs give a standard error of about sixteen points, which puts a rough ninety-five per cent interval at plus or minus thirty-one points. Forty runs narrows that to about plus or minus fifteen. The textbook approximation assumes independent runs, which consecutive queries in one session are probably not, so read these as the optimistic case.

The useful version of that: at ten runs you cannot tell one brand appearing a third of the time from one appearing two thirds of the time. So before anyone celebrates a month-on-month move, ask whether the move is wider than the band. A shift of five points on twenty prompts is inside the noise. A shift from one in five to three in five after a specific fix is worth a repeat run and a look at what changed.

How this differs from rank tracking

	Keyword rank tracking	AI visibility tracking
The unit	A position on a query	A ratio over prompts and runs
Day-to-day behaviour	Mostly steady	Moves between identical runs
Honest reporting format	A position, dated	A ratio plus its run count
Failure mode	Mistaking noise for a trend	Mistaking one reply for a finding
What fixes a drop	The page	Often somebody else's page

Choosing between a tool and a sheet comes down to how many runs you can afford and who has to read the output, which is the subject of our seven trial questions for a tracker.

Drop the composite score from the monthly report

The call: report detection rate with its run count, and leave the vendor's blended score out of the deck entirely. The reasoning is in the formula. A score that multiplies presence by placement, as Keyword.com's published formula does, cannot be diagnosed after the fact. When it falls, nobody in the room can say whether the brand stopped appearing or started appearing later in the answer, and those two problems have different fixes. A raw ratio, dated, with the number of runs beside it, carries less authority and far more information.

The fair counter-case is that leadership wants one comparable number, and a composite is how every other channel reports. True. But a composite is only comparable against itself, because weighting differs by vendor, so the comparison it appears to offer across tools is not real. A dated ratio with a sample size is the smaller promise that happens to be keepable.

The thing to wait for is a shared, published definition that several vendors compute identically, the way impressions and clicks became comparable once one platform defined them. If that lands, a composite becomes worth reporting, because it would then mean the same thing in two tools. Until then, the composite's main function is to make a volatile measurement look stable.

What repeated runs still will not tell you

Sampling fixes precision, not scope. It does not tell you how many buyers saw those answers, whether a mention moved a deal, or why one source got chosen over another. On the traffic side, the ceiling is low by design: in a March 2025 panel of 900 adults in the United States, Pew Research Center recorded a click on a link inside an AI summary during 1% of visits where a summary appeared. That is Google rather than a chat assistant, and it is more than a year old, but it is the best-documented figure available, and it says plainly that clicks will understate presence. Pair the rate with branded search volume and with what buyers tell you on calls.

Measure the distribution, not the screenshot

One answer is an anecdote with a timestamp. A rate across many prompts and several runs, logged with its date, its assistant and its session conditions, is a measurement. Build the prompt set once and keep it fixed, and make sure it is not stacked with your own brand name, which is what an unrigged prompt set is for. Then log it the cheap way every month. Variation is not a reason to stop measuring. It is the reason to report a sample size.