As AI search became something we were advising clients on, we realised we needed a way to actually check it, rather than relying on manual spot-checks across different AI tools. This is what we built, and what we learned putting it together.
The problem we were solving
Checking how a business is described or recommended across ChatGPT, Gemini, and other AI tools meant manually asking each one the same question and comparing answers, which does not scale past one or two checks a week.
What we built
A lightweight internal tool that sends a set of realistic customer queries, structured around how a real customer might ask for a recommendation, to multiple AI models via their APIs, then logs and compares the responses over time.
Query design mattered more than expected.
Early versions used queries that were too generic and returned inconsistent, low-signal results. Rewriting queries to match how real customers actually phrase requests, including location and specific needs, produced far more useful and consistent data.
Tracking change over time, not just a single snapshot.
A single check tells you very little. Running the same queries weekly and tracking whether a client starts appearing, stops appearing, or changes position in the response is what actually shows whether AEO work is having an effect.
What surprised us
Different AI models cited noticeably different sources for the same query, which confirmed that optimising for one AI tool does not guarantee visibility across all of them. Consistency of business information across the web mattered more than we initially expected, more than the quality of any single page.
How we use it now
This tool now runs as part of how we report on AEO work for clients, giving an actual before-and-after comparison rather than a vague claim that visibility has improved. It also flags early when a competitor starts appearing where a client previously held the answer alone.
What is next
We want to expand the query set to cover more realistic variations and add automatic alerts when a tracked client’s visibility changes meaningfully, rather than needing someone to check the dashboard manually.