Why AI Crawlers Are Draining Resources and Skewing Analytics Data

AI Brand Recommendations Change with Each Input: Insights on Consistency

Update Shifting Landscapes: The Variability of AI Brand Recommendations Recent research by SparkToro has shed light on a perplexing aspect of artificial intelligence tools: the inconsistency of brand recommendations. It reveals that generative AI platforms such as ChatGPT and Google’s AI search features return radically different outputs nearly every time they are queried, even with identical prompts. The study conducted by Rand Fishkin, SparkToro co-founder, alongside Patrick O’Donnell from Gumshoe.ai, found that less than 1% of repeated queries returned the same brand list. This staggering statistic raises questions about the reliability and practical use of AI recommendations in marketing and consumer decision-making. The Research: Methodology and Results To gather their data, the researchers ran 2,961 queries across three different AI platforms utilizing 600 volunteers. They tested 12 specific prompts, focusing on categories like chef’s knives and digital marketing consultants. The results were striking: each response varied not only in the brands listed but also in the order of those recommendations. Despite the diverse prompts, a handful of brands like Bose, Sony, and Apple frequently emerged, illustrating a degree of consistency in brand presence amidst the chaos. This highlights that while the exact outputs differ, the semantic landscape of brand consideration remains somewhat stable. Implications for Marketers: Rethinking AI Ranking Systems This study calls into question the approach many companies have taken towards “AI ranking position” as a meaningful metric. With the lack of repeatability, brands may be throwing money at tools that claim to track rankings in AI without sufficient evidence of their effectiveness. Fishkin pointed out, “any tool that gives a ‘ranking position in AI’ is full of baloney.” Instead, he suggests focusing on how often a brand appears across myriad prompts as a more reliable indicator of visibility. Understanding User Intent: The Role of Prompts An interesting facet of the findings is how real users craft their prompts. The diversity in how 142 participants approached a simple query about headphones leads to a semantic similarity score of only 0.081—much lower than expected. Fishkin used the analogy of “Kung Pao Chicken and Peanut Butter” to emphasize that although prompts can share core intent, they often diverge dramatically in content and structure. This variation further complicates the AI’s ability to provide consistent recommendations. Future Predictions: What's Next for AI and Brand Recommendations? The implications of this study stretch beyond just marketing budgets. As AI technology continues to evolve, understanding its limitations will be crucial for businesses. Companies should recalibrate their expectations surrounding AI-generated recommendations, embracing the chaos rather than relying on consistent outputs. Future advancements may focus on the creation of tools that can synthesize user intent more effectively while managing the inherent unpredictability of AI outputs. As we look towards 2025 and beyond, continued innovation in artificial intelligence will be essential, sparking new methodologies for interpreting AI data. Conclusion: Rethink AI Tools' Reliability As organizations look to leverage AI for marketing and product recommendations, it's clear that a shift in strategy is necessary. This research highlights the significance of focusing on brand visibility rather than rigid ranking systems. Businesses must adapt to this new reality, acknowledging that with AI, consistency may not be a guarantee but potential consumer insight remains ripe for exploration.