Anthropic’s Claude 3.5 Sonnet delivered the strongest overall performance in Galileo’s Hallucination Index, an evaluation of 22 generative-AI models used with retrieval-augmented generation. Google’s Gemini 1.5 Flash stood out for cost-effectiveness, while Alibaba’s Qwen2-72B-Instruct led the open-model group.
The index focused on a practical enterprise problem: whether a model stayed faithful to information supplied in its context. A fluent answer could still be wrong if it ignored, distorted or invented facts, making hallucination performance central to applications that depended on company documents or retrieved knowledge.
Galileo expanded the comparison by 11 models as both open and closed systems multiplied. The results showed no single winner across every purchasing consideration. Accuracy, context length, model size and price all influenced which system might be suitable for a particular deployment.
The index covered systems from OpenAI, Anthropic, Google and Meta alongside newer competitors. Its expanded field reflected how quickly enterprise buyers had moved from choosing among a few prominent chatbots to comparing a varied market of models with different costs and deployment options.
That diversity made a task-specific comparison more useful than assuming the most famous or largest model would automatically be the most reliable.
Claude Performed Consistently Across Context Lengths
Claude 3.5 Sonnet scored close to perfectly across short, medium and long-context scenarios, making it the strongest overall model in Galileo’s evaluation. Consistency mattered because retrieval systems could supply anything from a compact passage to a large collection of documents.
The test used input lengths ranging from 1,000 to 100,000 tokens. Galileo’s proprietary “context adherence” metric examined whether outputs remained grounded in that supplied material. A model that performed well only with short prompts would be less dependable when an enterprise asked it to analyse lengthy records or combine evidence across sources.
Claude’s result indicated that Anthropic had produced a strong balance across those conditions. It did not prove that the model would never hallucinate, and the metric belonged to Galileo rather than an independent universal standard. It did provide a comparative signal for organisations evaluating RAG systems under more realistic conditions than a narrow academic task.
Gemini 1.5 Flash Balanced Cost and Performance
Google’s Gemini 1.5 Flash was identified as the best model for cost-effectiveness while maintaining strong results across the tested tasks. That distinction was important because the most accurate system was not automatically the best operational choice at high query volumes.
An enterprise might accept a small difference in performance if a cheaper model made a service economically viable. Conversely, a high-stakes use could justify paying more for the strongest grounding. The index therefore treated model selection as a balance among reliability, performance and price rather than a single leaderboard.
Google’s results also showed a sharp difference inside its own model portfolio. Gemini 1.5 Flash ranked near the top, while the open Gemma-7B model performed poorly in the hallucination tests. Sharing a developer did not guarantee similar reliability; architecture, size, training and product design still mattered.
Open Models Were Closing the Reliability Gap
Alibaba’s Qwen2-72B-Instruct was the strongest open model, particularly in short and medium contexts. Mistral Large also contributed to a broader pattern of competitive systems emerging outside the United States.
Galileo found that open models were improving their hallucination performance while often costing less than closed alternatives. This could give enterprises more control over deployment and create competitive pressure on proprietary providers. The results did not show that every open model had caught up, as Gemma’s weak result made clear.
The comparison also challenged the assumption that larger always meant more reliable. Smaller systems sometimes beat larger ones, suggesting that efficient design and training could matter more than parameter count alone. For customers, that opened the possibility of reducing cost and infrastructure demands without automatically accepting poorer grounding.
Longer Contexts Improved Without Eliminating Risk
The index reported substantial progress in handling extended context lengths without a corresponding collapse in quality. That development supported more ambitious RAG applications, including analysis of long documents and larger internal knowledge collections.
Long context did not by itself solve hallucinations. A model still had to identify relevant evidence, follow instructions and avoid adding unsupported claims. More tokens increased the amount of material available, but could also make evaluation and retrieval more complex.
Galileo’s chief executive, Vikram Chatterji, argued that many existing benchmarks reflected academic use cases rather than the decisions enterprises faced. The Hallucination Index attempted to make the comparison more practical by testing context adherence, length and cost together.
The most useful conclusion was not that one provider had permanently defeated hallucinations. Claude 3.5 Sonnet led this test, Gemini 1.5 Flash offered the strongest price-performance balance, and Qwen2 led among open systems. Each result answered a different procurement question.
For organisations, the index supported a disciplined approach: define the context lengths and error tolerance of the intended application, compare models under those conditions and monitor unsupported outputs after deployment. Hallucinations remained a barrier to production AI, but the range of credible choices was expanding.
