# Is AI Visibility a Myth? The Staggering Inconsistency of LLM Brand Recommendations - Canonical: https://www.geoly.ai/blog/ai-visibility-myth-geo-strategy - Published: 2026-02-02 - Updated: 2026-07-04 - Author: [GEOly AI](https://www.geoly.ai/blog/author/geoly-ai) > **Summary:** Still reporting your ChatGPT ranking with a single screenshot? The data says you're doing it wrong. A joint study by SparkToro and Gumshoe shatters the illusion of stable AI rankings. We deconstruct the randomness of LLMs and show you how to master Generative Engine Optimization (GEO) using a Monte Carlo mindset ## TL;DR: The Key Takeaways - **Core Discovery**: A joint study by SparkToro and Gumshoe reveals that LLM brand recommendations are highly random. - **The Data**: In repeated tests using the exact same prompt, major models (ChatGPT, Claude, Gemini) showed **<1% complete consistency**. - **Strategic Pivot**: Marketers must stop tracking "ranking position" and start tracking "Share of Voice (SOV)" probability and brand co-occurrence. ## The Core Abstract: From Deterministic Rankings to Probabilistic Clouds In the traditional SEO era, search results were **deterministic**. If you Googled "best CRM software," the #1 result usually stayed stable for weeks or even months. This ingrained a deep intuition in us: visibility is linear, stable, and precisely measurable. However, as a GEO (Generative Engine Optimization) expert, I need to share a counter-intuitive truth: **In the world of Generative AI, "rankings" as we know them do not exist. Chasing a single "visibility spot" is like driving using only your rearview mirror.** The latest **SparkToro/Gumshoe study** reveals an unsettling reality: LLM brand recommendations are shockingly inconsistent. If you are still reporting "I rank #1 on ChatGPT" to your boss, you might just be getting fooled by random noise. ## The Evidence: Devastating Data from SparkToro/Gumshoe To test the stability of AI recommendations, SparkToro founder Rand Fishkin and the Gumshoe team conducted a massive experiment. They didn't just ask a question once; they used automated scripts to run the exact same prompts hundreds of times to quantify the variance in AI outputs. The results were **devastating**, shattering the traditional fantasy of "AI Rank Tracking": ### 1. 0.00% Complete Consistency In tests across **ChatGPT-4**, **Claude 3**, and **Google Gemini**, when asked the same brand recommendation question (e.g., "List the 5 best email marketing tools"), **almost no two responses were identical**. - **<1% Repetition**: In the vast majority of test groups, the statistical probability of getting the exact same list of brands twice is near zero. - **Randomized Ordering**: Even when the same brands were included, their order was almost never repeated. The difference between Position 1 and Position 5 might just be a random number seed. ### 2. Arbitrary List Lengths The AI couldn't even decide how many products to recommend. For the same standardized question, it might suggest 3 brands on the first run and 10 on the second. This volatility renders the concept of "Top 3" meaningless. ### 3. "Hallucinations" and "Amnesia" A brand hailed as an "industry leader" in Run #1 might completely vanish in Run #2, only to return as an irrelevant footnote in Run #3. > **Expert Insight**: If you see your brand in a ChatGPT answer this month and celebrate your "SEO strategy," you likely just got lucky and hit a positive sample in the LLM's probability distribution. Conversely, missing out doesn't mean your GEO strategy failed. ## Technical Deconstruction: Why Do LLMs Behave So unpredictably? To understand this inconsistency, we need to look under the hood. This isn't just an "algorithm update"; it's a paradigm shift from **Deterministic Retrieval** to **Probabilistic Generation**. ### 1. Temperature and Random Seeds LLM output is controlled by a "Temperature" parameter. - **The Mechanism**: Unless the temperature is set to 0 (and the model supports strict deterministic output, which is rare in commercial closed-source models), the model always introduces randomness when selecting the next Token (word/character). - **The Misconception**: AI is a database looking up the best answer in its index. - **The Reality**: AI is a reasoning engine rolling dice every time it generates text to decide which brand "looks" like the most plausible answer. ### 2. The Butterfly Effect of Context Windows The SparkToro study found that even slight variations in user prompts cause AI outputs to fluctuate wildly. Even with the exact same prompt, tiny state differences on the server side or hidden updates to the System Prompt can cause a "butterfly effect," completely changing the recommendation list. ## Strategy: The KPI Revolution in the GEO Era Admitting that "AI visibility is unstable" doesn't mean we give up on GEO. Instead, it requires us to shift our KPIs from **"Position"** to **"Share of Voice."** ### 1. Ditch the "Snapshot," Embrace "Monte Carlo Simulations" Since a single query is meaningless, we must measure statistically. - **Old Way (SEO)**: Search once, take a screenshot, put it in a PPT. - **New Way (GEO)**: Run the same prompt 100 times. - Brand A appeared 82 times = **82% Visibility Probability**. - Brand B appeared 15 times = **15% Visibility Probability**. - **Conclusion**: You can't guarantee you'll appear in a specific search, but you can work to increase your **probability** of appearing across 100 searches. ### 2. "Co-occurrence" > "Keyword Density" LLMs learn **semantic associations** between concepts. To increase your probability of being mentioned, your brand must frequently **co-occur** with "specific attributes" or "categories" in the training data. - **Action Plan**: Don't just claim "we are the best CRM" on your own website. - **External Validation**: You need third-party authority sites (reviews, forums, news) to frequently mention your brand when discussing "efficient sales tools." This increases the odds that the model will "associate" your brand via probabilistic connection when generating content about "sales tools." ### 3. Beware of "Decision-Stage Substitution" This is another counter-intuitive trap. Studies show a brand might have high visibility in "informational" queries (e.g., "What is a CRM?"), but be swapped out in "transactional" queries (e.g., "Which CRM should I buy?"). - **Strategy Adjustment**: Don't just optimize for general content; build a corpus specifically for **comparative (Vs.)** and **superlative (Best)** contexts. ## Conclusion: Embrace the Chaos "AI Visibility" as a static metric is a **myth**. But as a **probability distribution function**, it is very real. The SparkToro/Gumshoe study isn't telling us to give up. It's warning us: **In this new era, marketing is no longer about renting a fixed billboard. It's about pouring enough dye into the ocean to statistically ensure the water eventually turns your color.** Stop tracking rankings. Start tracking probabilities.