Forge Mark

How an AI answer is checked and counted

6 minute read

Five assistants are asked the questions your shoppers ask, and every answer is read for who gets named. Follow one question from the asking to the weekly trend, and see why a single run can't be trusted.

The question

Every check starts with a question a buyer would really ask. Not a keyword. A sentence, the way someone types it into an assistant: "Which waterproof jacket is best for dog walking under £150?" or "Who fits kitchens in Leeds?" Those are samples. Yours come from your own site and from what people search for.

Forge Mark's tracked questions follow what people search, using your Search Console data, so the list isn't a guess about what buyers might ask. It's a list of what they're already looking for.

The engine

The same question goes to each assistant: ChatGPT, Gemini, Google AI Overviews and AI Mode, and Perplexity. Claude is switched on site by site. Each is asked the way a shopper in your country would ask, because an answer in London isn't the answer in Toronto.

Asked as a shopper in the UKWeekly check, 12 Oct

which waterproof jackets actually stay dry in heavy rain
  1. ChatGPTChatGPT

    #3named

  2. GoogleAI Overviews

    Not named

  3. Gemini

    Not named

  4. Perplexity

    #1named

  5. ClaudeClaude

    Not named

22%of answers

Share of answers naming Northfield

Up from 18%

8 weekly checks22%

Illustration on sample data.

Each assistant gives its own answer, with its own favourites. You can be named by one and missed by another, so the check keeps them apart and puts them side by side.

Parsing

An answer is just text. Before you can count anything, it has to be read. Forge Mark uses a Claude model to read each answer and pull out three things.

  • Who is named. Your brand, and every other brand in the answer.
  • In what place. First in a list of five is not the same as fifth.
  • Which pages are cited. The links the assistant gives as its sources.

It reads the words the assistant wrote. It doesn't need access to your site or to anything private.

Counting

Once each answer is read, the counting is simple.

  • Share of answers is the share of answers that name you. If you appear in 6 of 10, that's 60%.
  • Place is where you're named in the answer: first, second, last.
  • Rivals are the brands named instead of you. Forge Mark finds them from the brands the answers keep naming, so you don't have to list your competitors.
  • Cited pages are the pages from your site that each assistant links to.

Alongside each AI Overview, Forge Mark shows Google's organic top 10 for the same question. That shows whether the assistant is repeating the search results or going somewhere else.

The weekly trend

A single check tells you where you stand today. It can't show if you're improving. So Forge Mark asks every week, and draws a line.

The line is what to watch. A rise after you've published a page or fixed a technical problem is a sign it worked. A fall after an assistant changes its model is a sign something outside your control moved. Either way, you see it the week it happens, and not by chance months later.

Why one run is noisy

Ask an assistant the same question twice and you may get two different answers. They don't look things up the way a search engine does. They generate a reply each time.

So one run is a sample, not a measurement. Suppose you're named in 6 of 10 runs. That's 60%, but with only ten runs the real figure could plausibly be anywhere from about 30% to about 85%. Those are sample numbers, and they show the width of the problem.

Two things shrink the noise. More runs of each question, and more questions. Averaging over many questions and many weeks gives a figure you can lean on, which is why Forge Mark reports a trend and not a single day's answer.

Citations and mentions

A mention is when an assistant names your brand in its answer. A citation is when it links to one of your pages as a source. They aren't the same.

  • You can be mentioned without being cited. The assistant knows of you from elsewhere and names you, but links to someone else's review.
  • You can be cited without being mentioned. The assistant uses your page to answer, but names a rival.

Mentions measure how well known you are. Citations show which of your pages the assistants trust. Both matter, and the cited pages are the ones to look after first.

Gaps and fixes

A gap is a question where no assistant names you. Forge Mark lists every one of them. Each gap comes with a brief for the fix, and a test of whether the fix worked.

It also groups questions into topics, so you can see where to win next, and runs technical checks on your site. It lists the sources that assistants draw on for your market, which is where digital PR can help, and keeps a context bank.

Take the sample question about waterproof jackets. If every assistant answers it and none names you, that's a gap. The brief might say to add a page that answers it directly, with the price, the sizes and a comparison. A few weeks later the test asks again and shows if you're now named.

The fix itself is yours to make, or you can send it to Forge as a costed job.

What it can't tell you

  • How many people asked. Assistants don't publish how often a question is put. The search data behind the tracked questions is the closest guide.
  • What a particular person saw. Answers can differ by person and by day. A check is a sample of what an assistant says.
  • Private conversations. Forge Mark asks its own questions. It doesn't see anyone's chats.
  • Why an assistant chose a rival. It shows who was named and what was cited. The reason is yours to work out.

Visits and sales that really arrive from an assistant are measured separately by the pixel, in Forge Mark's Shoppers screens.

Frequently asked questions

Which AI assistants does Forge Mark ask?

Every week it asks ChatGPT, Gemini, Google AI Overviews and AI Mode, and Perplexity. Claude is switched on site by site.

Why do my results change from week to week?

Assistants write a fresh answer each time, and they change their models. That's why Forge Mark reports a trend across many questions rather than one day's answer.

What's the difference between a mention and a citation?

A mention is your brand named in the answer. A citation is one of your pages linked as a source. You can have either without the other.

Do I choose the questions?

Forge Mark writes them from your site's public pages and follows what people search, from your Search Console. It writes the questions with AI, and a new account can do that three times.

See how assistants answer for your shop.

Connect your site in a few minutes and see which of your shoppers' questions name you, and which name someone else.