Note

How often AI looks up your brand, and what it says when it doesn't

Split illustration. A speech bubble at the top asks, What do you know about [your brand]? On the left, a line runs from the bubble to a stack of web pages labelled Looked it up today. On the right, a line runs to a dusty filing cabinet with an open drawer, labelled Remembered from last year. Text across the bottom reads, Same question. Two different answers.

When someone asks an AI assistant about your company, the answer comes from one of two places. Either the assistant runs a web search right then and builds the answer from what it finds, or it answers from what the model learned in training, which can be a year or more out of date.

Two years ago search was an add-on. Today ChatGPT, Gemini and Claude all decide for themselves when to look something up, Perplexity searches on every question, and Google's AI Overviews, built on its search index, reach more than 2.5 billion people a month.

We went through the research on how often these assistants search, and for which questions. Then we started testing it in our own audits. This post covers both, and the changes we made to our method because of them.

What ChatGPT searches for

Most of the testing has been on ChatGPT, so we start there. The numbers come from small tests run by SEO and analytics firms, so treat them as rough.

Bar chart of how often ChatGPT searched the web by type of question in small vendor tests: close to 100% for questions about a specific company, its pricing or what it does; 58% to 87% for buying questions such as best X for Y and comparisons, depending on the test; and 0.9% for general knowledge questions such as what is X or how does X work.

A buyer early in their research often starts with a general question about the category. If those questions behave like other general questions, ChatGPT answers them from training data, and the brands it names are the ones the model already knew about.

What Claude, Gemini and Google search for

The other assistants have been measured less. The data that exists shows the same thing: how often they search depends on the question.

Claude. LLM Pulse asked the Claude app 400 buying questions 18 times each at the start of November. Claude's Sonnet model searched in 56.6% of answers and its Haiku model in 61.0%. By category the spread was much wider. Claude searched every time on wireless earbuds, robot vacuums and smartphones, and 1.4% of the time on skincare and language apps. Two tests by Profound landed far apart for the same reason: 36.6% on a mix of recommendation and explainer prompts, and 93% on a different set.

Gemini. cloro tracks how often each assistant's answers carry at least one live source, using its customers' prompts, which lean commercial. In October, 68.2% of the Gemini app's answers did, against 97.6% for ChatGPT and 99.9% for Perplexity. Gemini was lowest on all three of cloro's readings, and its monthly rate this year has swung between 27% and 70%.

Google's AI Overviews. An Overview is always built from search, so the number to look at is how often one appears. Trackers put that at roughly 43% to 48% of Google queries this summer (Similarweb and Semrush), up from 13% to 20% a year before. Researchers at Washington University in St. Louis found an Overview on 64.7% of trending queries phrased as questions and 9.5% of the rest. In cloro's data, an Overview appeared on 87.5% of brand queries and 97.6% of product queries.

These are vendor tests and one academic study, each with its own prompts, so the numbers can't be compared directly. What they share is that on every assistant, some questions about your category get a searched answer and some get an answer from memory. From the outside you cannot tell which one a buyer got.

ChatGPT is searching less overall

Semrush clickstream data put ChatGPT's overall search rate at about 46% of queries worldwide in late 2024 and 34.5% in February 2026. That second figure is US-only, and we have seen it only in secondhand summaries, so treat the comparison loosely.

The likely reason is speed. ChatGPT's fast default mode answers more from memory. In a June test by Semrush and Kevin Indig, the same model searched about 4.6 times as often when set to think harder, and the two settings shared only about a quarter of the sites they cited. Two buyers asking the same question can get answers built from different sites, depending on which mode they are in.

API answers and app answers do not match

Most AI visibility tracking runs questions through the assistants' developer APIs, because that is cheap and repeatable. A Surfer study from August ran 1,000 prompts both ways across five AI engines. Only 21% to 32% of the brands named in the API answers were also named in the consumer apps. ChatGPT's API named 13.8 brands per answer and the app named 7.9. Each prompt ran once each way, so part of that gap is ordinary randomness.

One run of a question is also a weak measure. Rand Fishkin and Gumshoe found under a 1 in 100 chance of an assistant returning the same list of brands twice. A brand's position in a single list changes almost every time. The share of runs it appears in is steady enough to report.

What we are seeing in our own runs

This month we started running our audit questions both ways, with search on and with search off, several times each. Here is what we have seen so far.

Our old method could not show any of this.

Why we are changing our method

Until now, our audits asked each question once per assistant, and whether search was on depended on the platform. That is a common setup, and it blends the searched answer and the memory answer into one number. Our runs show the two can differ on the same question, and that they change for different reasons.

We now measure three things:

  1. What buyers see. We run the questions in the consumer apps, in their default mode, and log on every run whether the assistant searched and what it cited.
  2. What the model remembers. We run the same questions with search turned off. Buyers get this answer on the questions that never trigger a search, and it changes only when a new model ships.
  3. How often, across several runs. Each question runs several times in each mode. We report how often a brand is named, with a range, in place of where it ranked in a single answer.

The gap between the searched answer and the memory answer tells you what kind of work is needed.

If a brand shows up when the assistant searches and is missing from memory, it is missing on the general questions buyers ask first. That gets fixed slowly, because memory only updates with a new model.

If the model remembers a brand well and search no longer backs that up, competitors are winning the searched answers today. That can be worked on this quarter.

If the two describe the brand differently, the model learned an older version of the company. It will keep saying so until the next release, however good this quarter's coverage is.

Search results change weekly. Memory changes with each model.

SISTRIX tracked more than 80,000 prompts every week for 17 weeks. For the typical prompt, no single source stayed in ChatGPT's searched answer for the whole period.

Most of that churn reverses. Ahrefs found that about three quarters of citations that drop out come back within 20 days.

Some of it sticks. In August, right after OpenAI swapped ChatGPT's default model and changed how it searches, Reddit's share of ChatGPT's citations fell from 3.83% to 0.52% in about a week, by Promptwatch's count. In our skincare study, sources that had led the category in the spring did not appear at all. A brand reviewing its AI presence once a quarter could have gone three months without knowing.

Memory changes on a different schedule. When Anthropic released Claude Fable 5.2 on November 9, what Claude says about a brand from memory changed that day, for every brand, whether or not anything about the brand had changed.

So we are adding a monthly program alongside the quarterly one:

We are not building a dashboard. Most of what moves week to week reverses on its own, so the monthly read reports the changes that stuck and what to do about them.

One related change from this week. On November 10 OpenAI opened Businesses in ChatGPT, in beta. An eligible company can claim its profile, correct its facts, and see how often ChatGPT mentions it and how many clicks follow. If you can claim yours, do. It is a useful signal between measurements. It covers one assistant and counts mentions, so we use it alongside the measures above.

If you want to see how this looks for your brand, a one-page snapshot is free. Every snapshot from here on uses the new method. Here is a sample, built on a made-up meal kit brand.

Sample one-page AI snapshot for Fernhill Kitchen, a made-up meal kit brand. The left column lists four findings: named in 93 of 144 answers with search on, named in 12 of 144 with search off, memory describing the company as it was two years ago, and all three runs agreeing on 7 of 12 questions. Below them are three things to work on. The right column has a table of how often each competitor is named with search on and search off, and how often each assistant searched. A note at the bottom says every name and number on the page is made up.
A sample. The brand, its competitors and every number are made up to show the format.
← All posts