TL;DR: The Key Takeaways
- Core Discovery: A joint study by SparkToro and Gumshoe reveals that LLM brand recommendations are highly random.
- The Data: In repeated tests using the exact same prompt, major models (ChatGPT, Claude, Gemini) showed <1% complete consistency.
- Strategic Pivot: Marketers must stop tracking "ranking position" and start tracking "Share of Voice (SOV)" probability and brand co-occurrence.
The Core Abstract: From Deterministic Rankings to Probabilistic Clouds
In the traditional SEO era, search results were deterministic. If you Googled "best CRM software," the #1 result usually stayed stable for weeks or even months. This ingrained a deep intuition in us: visibility is linear, stable, and precisely measurable.
However, as a GEO (Generative Engine Optimization) expert, I need to share a counter-intuitive truth: In the world of Generative AI, "rankings" as we know them do not exist. Chasing a single "visibility spot" is like driving using only your rearview mirror.
The latest SparkToro/Gumshoe study reveals an unsettling reality: LLM brand recommendations are shockingly inconsistent. If you are still reporting "I rank #1 on ChatGPT" to your boss, you might just be getting fooled by random noise.
The Evidence: Devastating Data from SparkToro/Gumshoe
To test the stability of AI recommendations, SparkToro founder Rand Fishkin and the Gumshoe team conducted a massive experiment. They didn't just ask a question once; they used automated scripts to run the exact same prompts hundreds of times to quantify the variance in AI outputs.
The results were devastating, shattering the traditional fantasy of "AI Rank Tracking":
1. 0.00% Complete Consistency
In tests across ChatGPT-4, Claude 3, and Google Gemini, when asked the same brand recommendation question (e.g., "List the 5 best email marketing tools"), almost no two responses were identical.
- <1% Repetition: In the vast majority of test groups, the statistical probability of getting the exact same list of brands twice is near zero.
- Randomized Ordering: Even when the same brands were included, their order was almost never repeated. The difference between Position 1 and Position 5 might just be a random number seed.
2. Arbitrary List Lengths
The AI couldn't even decide how many products to recommend. For the same standardized question, it might suggest 3 brands on the first run and 10 on the second. This volatility renders the concept of "Top 3" meaningless.
3. "Hallucinations" and "Amnesia"
A brand hailed as an "industry leader" in Run #1 might completely vanish in Run #2, only to return as an irrelevant footnote in Run #3.
Expert Insight: If you see your brand in a ChatGPT answer this month and celebrate your "SEO strategy," you likely just got lucky and hit a positive sample in the LLM's probability distribution. Conversely, missing out doesn't mean your GEO strategy failed.
Technical Deconstruction: Why Do LLMs Behave So unpredictably?
To understand this inconsistency, we need to look under the hood. This isn't just an "algorithm update"; it's a paradigm shift from Deterministic Retrieval to Probabilistic Generation.
1. Temperature and Random Seeds
LLM output is controlled by a "Temperature" parameter.
- The Mechanism: Unless the temperature is set to 0 (and the model supports strict deterministic output, which is rare in commercial closed-source models), the model always introduces randomness when selecting the next Token (word/character).
- The Misconception: AI is a database looking up the best answer in its index.
- The Reality: AI is a reasoning engine rolling dice every time it generates text to decide which brand "looks" like the most plausible answer.
2. The Butterfly Effect of Context Windows
The SparkToro study found that even slight variations in user prompts cause AI outputs to fluctuate wildly. Even with the exact same prompt, tiny state differences on the server side or hidden updates to the System Prompt can cause a "butterfly effect," completely changing the recommendation list.
Strategy: The KPI Revolution in the GEO Era
Admitting that "AI visibility is unstable" doesn't mean we give up on GEO. Instead, it requires us to shift our KPIs from "Position" to "Share of Voice."
1. Ditch the "Snapshot," Embrace "Monte Carlo Simulations"
Since a single query is meaningless, we must measure statistically.
- Old Way (SEO): Search once, take a screenshot, put it in a PPT.
- New Way (GEO): Run the same prompt 100 times.
- Brand A appeared 82 times = 82% Visibility Probability.
- Brand B appeared 15 times = 15% Visibility Probability.
- Conclusion: You can't guarantee you'll appear in a specific search, but you can work to increase your probability of appearing across 100 searches.
2. "Co-occurrence" > "Keyword Density"
LLMs learn semantic associations between concepts. To increase your probability of being mentioned, your brand must frequently co-occur with "specific attributes" or "categories" in the training data.
- Action Plan: Don't just claim "we are the best CRM" on your own website.
- External Validation: You need third-party authority sites (reviews, forums, news) to frequently mention your brand when discussing "efficient sales tools." This increases the odds that the model will "associate" your brand via probabilistic connection when generating content about "sales tools."
3. Beware of "Decision-Stage Substitution"
This is another counter-intuitive trap. Studies show a brand might have high visibility in "informational" queries (e.g., "What is a CRM?"), but be swapped out in "transactional" queries (e.g., "Which CRM should I buy?").
- Strategy Adjustment: Don't just optimize for general content; build a corpus specifically for comparative (Vs.) and superlative (Best) contexts.
Conclusion: Embrace the Chaos
"AI Visibility" as a static metric is a myth. But as a probability distribution function, it is very real.
The SparkToro/Gumshoe study isn't telling us to give up. It's warning us: In this new era, marketing is no longer about renting a fixed billboard. It's about pouring enough dye into the ocean to statistically ensure the water eventually turns your color.
Stop tracking rankings. Start tracking probabilities.