Microsoft’s latest diagnostic study turns the spotlight toward sequential medical reasoning. TechCrunch reported the Science publication on October 22.
The relevant question is not simply whether an AI can name a disease from a finished case description. It is how the system gathers information, what it asks for, and how researchers decide whether its final answer is right. The public technical documentation makes those evaluation choices unusually important to interpreting the announcement.
A benchmark built around successive decisions
Microsoft’s October 22 research announcement describes SDBench, a sequential diagnosis benchmark, and MAI-DxO, a system coordinating a panel of AI agents. It says the evaluation includes 304 complex New England Journal of Medicine cases and a separate set of emergency-department presentations from MIMIC-IV-ED.
The company reports 83% diagnostic accuracy on the complex cases, compared with 36% for physicians using reference resources and 22% without them. On the emergency-department cases, it reports 80% for the system and 47% for physicians. These are Microsoft’s reported benchmark results, not outcomes from treating patients in ordinary clinical practice.
The comparison needs to remain attached to its setting. A percentage on one case collection does not describe performance across every specialty, hospital or patient population. Nor does a benchmark accuracy measure, on its own, show whether deploying the system would improve care within a working health service.
The gatekeeper and judge shape the result
The researchers’ original SDBench documentation explains three roles: a diagnostic agent, a gatekeeper supplying requested information, and a judge scoring the final diagnosis. The gatekeeper returns recorded results when available, but can provide realistic synthetic results when a case does not contain the requested information. The judge uses a five-point scale, with scores of four or five counted as correct.
That design makes the experiment more interactive than a static question, while introducing evaluation machinery that readers need to understand. A generated test result is not a newly observed laboratory result from a patient. The scoring process is also a defined research procedure, not automatically equivalent to a clinician’s assessment of safety or usefulness.
The documentation further says that NEJM data requires licensing and MIMIC access requires credentialing, training and a data-use agreement. Five bundled synthetic cases are intended for development and testing, not reporting diagnostic performance. Availability of code therefore should not be confused with unrestricted access to all the clinical evaluation material or with reproducing the headline result on the default examples.
Clinical evaluation is the next test
Microsoft also announces a planned prospective evaluation with Beth Israel Deaconess Medical Center, involving direct patient interaction and comparison with physicians’ independent assessments. In the company’s account, that work is intended to examine performance across clinical settings and fit within real workflows. It is a future evaluation, not evidence that the benchmark’s advantages have already been demonstrated there. Source: Microsoft research announcement.
The distinction matters because an evaluation can succeed at one question while leaving another unanswered. Selecting the correct diagnosis from retrospective material is different from managing uncertainty when information is incomplete, a patient needs attention, and clinical responsibility remains with a care team. A prospective comparison would also need to distinguish diagnostic accuracy from practical usefulness. Even a correct suggestion has to arrive at a useful time, with enough supporting information for the responsible clinician to assess it and decide what to do next.
An informative next report would make those boundaries explicit: which patients and settings were included, what the system was permitted to do, how disagreements were handled, and what outcomes were measured. Until such evidence is available, the announcement is a research milestone with a documented evaluation framework. It is not a general demonstration that autonomous AI can replace clinical teams.
