TL;DR: The Key Takeaways
- Core Discovery: A joint study by SparkToro and Gumshoe reveals that LLM brand recommendations are highly random.
- The Data: In repeated tests using the exact same prompt, major models (ChatGPT, Claude, Gemini) showed <1% complete consistency.
- Strategic Pivot: Marketers must stop tracking "ranking position" and start tracking "Share of Voice (SOV)" probability and brand co-occurrence.
The Core Abstract: From Deterministic Rankings to Probabilistic Clouds
In the traditional SEO era, search results were deterministic. If you Googled "best CRM software," the #1 result usually stayed stable for weeks or even months. This ingrained a deep intuition in us: visibility is linear, stable, and precisely measurable.
However, as a GEO (Generative Engine Optimization) expert, I need to share a counter-intuitive truth: In the world of Generative AI, "rankings" as we know them do not exist. Chasing a single "visibility spot" is like driving using only your rearview mirror.
The latest SparkToro/Gumshoe study reveals an unsettling reality: LLM brand recommendations are shockingly inconsistent. If you are still reporting "I rank #1 on ChatGPT" to your boss, you might just be getting fooled by random noise.
The Evidence: Devastating Data from SparkToro/Gumshoe
To test the stability of AI recommendations, SparkToro founder Rand Fishkin and the Gumshoe team conducted a massive experiment. They didn't just ask a question once; they used automated scripts to run the exact same prompts hundreds of times to quantify the variance in AI outputs.
The results were devastating, shattering the traditional fantasy of "AI Rank Tracking":
1. 0.00% Complete Consistency
In tests across ChatGPT-4, Claude 3, and Google Gemini, when asked the same brand recommendation question (e.g., "List the 5 best email marketing tools"), almost no two responses were identical.
- <1% Repetition: In the vast majority of test groups, the statistical probability of getting the exact same list of brands twice is near zero.
- Randomized Ordering: Even when the same brands were included, their order was almost never repeated. The difference between Position 1 and Position 5 might just be a random number seed.
2. Arbitrary List Lengths
The AI couldn't even decide how many products to recommend. For the same standardized question, it might suggest 3 brands on the first run and 10 on the second. This volatility renders the concept of "Top 3" meaningless.
