AI Visibility Tracking: Establish a Baseline and Handle Variance
Build a reliable AI visibility baseline, handle failed runs and answer variability, and compare stable cohorts before reacting to a score change.
The short answer
Establish an AI visibility baseline by freezing a relevant prompt cohort and recording repeated valid observations under a documented protocol. Save provider, surface, locale, prompt version, and run status. Compare like-for-like samples and show variation as well as averages. A one-off answer is a snapshot; a reliable trend requires comparable questions and enough repeated evidence to justify the decision you intend to make.
Choose what your baseline is meant to detect
Define the event of interest before collecting data: any brand mention, an owned-site citation, the intended page being cited, or an accurate answer to a particular question. Each baseline serves a different purpose. If the team changes definitions after a release, it may mistake a reporting change for a visibility change.
Use a prompt cohort tied to an audience and product scope. Record the cohort size, observed providers, interface, locale, and schedule. Keep a stable core while placing new questions in an expansion cohort. You can learn from additional prompts without losing the comparability of the original panel.
Preserve observations and their failure states
Save each answer rather than only the daily score. Repeated observations reveal whether a prompt consistently cites your site or alternates among several sources. Where available, preserve the provider's model identifier and retrieval context. Consumer-interface observations and API outputs should be labeled separately because their surrounding conditions may differ.
Record timeouts, unavailable providers, and parsing failures as missing measurements. If a parser cannot extract links, do not label the answer as having no citations until a review confirms that. Track the missing-data rate as a guardrail. A trend produced by better extraction or fewer failed calls is an instrumentation change that deserves its own explanation.
Compare on a common cohort
Suppose a hypothetical panel schedules 60 checks each week. In week one, 50 are valid and 15 include your brand. In week two, 60 are valid and 18 include it. Both observed mention rates are 30%, although the raw mention count increased. Showing only the count would exaggerate the change; showing the valid denominator makes it interpretable.
For a stronger comparison, restrict both periods to the same prompt-provider combinations and disclose exclusions. Examine the distribution across prompts instead of only the total. If one popular prompt improved while several buying questions weakened, the average can hide an important business gap. Use intent and provider breakdowns to determine whether further investigation is justified.
Handle variance without pretending to know certainty
Use repeated runs to describe observed stability. For example, report that a page was cited in two of three checks for a given prompt rather than declaring it permanently present. Larger samples may help estimate variability, but repetition alone does not make a nonrepresentative prompt panel a measure of total market demand.
Set review rules based on the decision's cost. A small fluctuation may warrant another observation; a persistent loss across a stable cohort may justify inspecting page access and source freshness. Keep the rule documented and revise it as you learn. Avoid a universal alert threshold that ignores provider variability, missing observations, and the importance of the affected questions.
| Field | Example purpose |
|---|---|
| Stable cohort size | Shows which questions are truly comparable |
| Valid observations | Supplies the rate's denominator |
| Missing measurements | Reveals reliability problems |
| Repeated appearances | Describes observed variability |
| Intent breakdown | Shows whether commercially relevant questions changed |
Your next steps
- Write the visibility event and counting rule before the baseline run.
- Freeze the core prompt cohort and version all later edits.
- Preserve raw answers and failure states for every observation.
- Compare common prompt-provider combinations with valid denominators.
- Investigate persistent, relevant patterns before editing pages.
Common questions
How many repetitions are enough?
There is no universal number for every decision. Choose a protocol you can sustain, inspect the observed variability, and increase sampling when the cost of a wrong conclusion is high. Describe the sample honestly rather than presenting an arbitrary repeat count as statistical proof.
Can our score rise even when nothing changed on the site?
Yes. Answers, provider conditions, prompt composition, and measurement availability can change. Check those explanations first. A stable cohort and saved observations help separate a possible site effect from normal answer variation or a reporting change.
Sources & further reading
Official documentation supporting this guide. Worked examples are illustrative unless explicitly identified as measured results.