Google Assistant led Loup Ventures’ virtual-assistant IQ test, while Siri and Alexa showed strong progress. Each assistant received 800 questions covering common tasks, separating whether the system understood a request from whether it produced a correct or sufficient answer.
Understanding scores were nearly perfect: Google Assistant reached 100 percent, Alexa 99.9 percent and Siri 99.8 percent. Correct-answer results differed more sharply at 92.9 percent for Google, 83.1 percent for Siri and 79.8 percent for Alexa.
The questions covered local search, commerce, navigation, information and commands. This range attempted to test usefulness across daily assistant tasks rather than concentrate on a single trivia or transcription capability.
All three improved from 2018, when Google scored 86 percent, Siri 79 percent and Alexa 61 percent on correct responses. Microsoft’s Cortana was removed after the company shifted away from direct smart-speaker competition toward an embedded assistant strategy.
The announcement is best understood in its original publication context. It described the evidence, expectations and constraints visible at that time; later outcomes should not be read backwards into claims that had not yet been tested.
A useful assessment separates the demonstrated capability from the surrounding projection. It asks what changed for users or organisations, which dependencies sat behind the headline, and what evidence would justify expansion, redesign or withdrawal.
The comparison point also matters. A faster model, larger system or new partnership should be measured against the process it replaces, including waiting time, manual work, error correction and infrastructure already in place. Without that baseline, an impressive technical figure can conceal transferred cost or a problem that has merely moved to another part of the workflow.
Long-term performance requires more than a successful launch. Models, datasets, organisations and external rules change, while people learn to use a capability in ways its designers did not predict. Version records, periodic testing and a named operational owner create a way to detect drift, investigate incidents and decide whether the original case for deployment still holds.
Teams should also examine distribution. An average improvement may leave particular sites, languages, workloads or groups with a predictable weakness. Breaking results down by relevant use case, documenting corrections and inviting feedback from people closest to the process gives decision-makers a clearer view of where the system helps and where additional safeguards or a different approach are needed.
Understanding Was Not the Same as Answering
All three assistants recognised nearly every question, but their correct-answer scores remained separated by more than thirteen percentage points. The gap showed that speech recognition was only the first stage of a useful response.
The practical importance was that evaluation needs to follow interpretation, retrieval and action instead of treating successful transcription as task completion. A credible evaluation would connect the claim to representative outcomes, record exceptions and human intervention, and identify who could act when performance or conditions changed. It should also compare different sites, workloads or user groups and document the resources required to sustain the result. That approach prevents a strong demonstration or percentage from becoming a substitute for operational evidence.
Five Categories Broadened the Question Set
Local queries, shopping, navigation, factual information and commands exercised different data sources and actions. An assistant could perform strongly in one category while failing when a request required location, timing or an external service.
The practical importance was that category-level results help users and developers see weaknesses hidden by one overall percentage. A credible evaluation would connect the claim to representative outcomes, record exceptions and human intervention, and identify who could act when performance or conditions changed. It should also compare different sites, workloads or user groups and document the resources required to sustain the result. That approach prevents a strong demonstration or percentage from becoming a substitute for operational evidence.
Alexa Recorded the Largest Annual Gain
Alexa rose from 61 percent correct in 2018 to almost 80 percent, a larger improvement than its rivals despite finishing third. The result illustrated why both current rank and rate of progress matter when judging a competitive market.
The practical importance was that repeated benchmarks are most informative when questions and scoring remain comparable and changes are documented. A credible evaluation would connect the claim to representative outcomes, record exceptions and human intervention, and identify who could act when performance or conditions changed. It should also compare different sites, workloads or user groups and document the resources required to sustain the result. That approach prevents a strong demonstration or percentage from becoming a substitute for operational evidence.
Cortana’s Exit Reflected a Strategy Change
Microsoft repositioned Cortana as a capability embedded in other services instead of competing directly in consumer smart speakers. Excluding it prevented the test from comparing products aimed at increasingly different roles.
The practical importance was that benchmark participation should follow intended use; absence can reflect strategic focus rather than purely technical weakness. A credible evaluation would connect the claim to representative outcomes, record exceptions and human intervention, and identify who could act when performance or conditions changed. It should also compare different sites, workloads or user groups and document the resources required to sustain the result. That approach prevents a strong demonstration or percentage from becoming a substitute for operational evidence.
Across the development, technical performance was only one part of the system. Data access, software integration, skills, cost, governance and affected people could each determine whether the promised value survived outside the initial announcement.
The durable lesson is to preserve traceability and choice. Organisations should document assumptions, keep important outputs reviewable, monitor downstream effects and retain a realistic exit route. Those controls make it possible to learn from an ambitious AI initiative without turning early momentum into an irreversible commitment.
