AI Beats Doctors? Microsoft’s Paper Offers A More Specific Measure

🔍 Read the full analysis: AI Beats Doctors? Microsoft’s Paper Offers A More Specific Measure on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO system scored 83.2% on a set of 304 complex medical cases, compared with 36.1% for generalist physicians using reference materials, according to a peer-reviewed paper published Oct. 22 in Science. The results describe performance on historical, text-only cases in a simulator—not diagnoses made for real patients—and do not compare the system with specialist teams.

The paper, Sequential diagnosis with language models, also reports results from 325 emergency department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. Microsoft says its system uses a language model to coordinate a panel of virtual physicians. They suggest diagnoses, request additional case information or tests, and evaluate one another’s reasoning over multiple rounds.

The benchmark begins with a short case summary. Each additional detail must be requested from a simulated gatekeeper, and tests carry estimated costs. The system is assessed both on its final diagnosis and on the cost of reaching it. This differs from a conventional exam-style benchmark in which all case facts are provided at once.

The paper’s results are Microsoft’s findings about its own system. The source report says all 15 paper authors are affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. The paper was peer-reviewed, but that process does not establish that independent researchers have reproduced its results. The full Science paper is behind a paywall, and some methodological details described in the source report come from an earlier preprint, so they may not fully match the final publication.

A Score Is Not Clinical Proof

The study tests a clinically relevant question: whether an AI can gather information and select tests in a sequence, rather than simply choose an answer from a complete case file. That makes the benchmark a useful step beyond static medical-question tests. The cost estimates also reflect a practical concern for hospitals: diagnostic accuracy is not the only factor when a system requests tests.

But the scores do not show that MAI-DxO is ready to replace or outperform doctors in routine care. The comparison was between an AI system and individual generalists working alone, while the cases were selected for difficulty. The source report notes that such cases may be referred to specialists or handled by physician teams in clinical practice. The benchmark also used no real patients, physical examinations or imaging.

For patients, hospitals and insurers, the results are best read as evidence of strong performance on a defined test—not evidence of improved health outcomes, lower real-world spending or safer care. Those questions require studies in clinical settings, with appropriate comparisons and measures of harms as well as correct diagnoses.

How the Benchmark Was Built

Microsoft’s Sequential Diagnosis Benchmark is designed to model the information-gathering process. A participant starts with a brief presentation and decides which question or test to request next. A gatekeeper model holds the rest of the case information. According to the source report’s account of the earlier preprint, when a requested result was absent from the record, the gatekeeper generated a synthetic result rather than revealing that the information was unavailable.

The earlier preprint also described a model-based judge that rated diagnoses against the published final diagnosis. Physicians who re-graded a sample generally agreed with the judge, but not perfectly. Test prices were estimated using billing codes and a 2023 U.S. hospital price list, with a fixed $300 per physician visit; the authors characterized these figures as approximations. These details are reported from the preprint, and the final paper may differ.

The physician comparison also has limits. The source report says the preprint involved 21 U.S. and UK generalists—17 in primary care and four hospital generalists—and that doctors could not use search engines or language models. The Science paper reports an additional reference-aided group, with a 36.1% score. The tested NEJM cases contained difficult diagnoses and no healthy patients or harmless explanations, so the study could not measure false alarms across a representative patient population.

“MAI-DxO reached 83.2% diagnostic accuracy on 304 complex cases.”

— Microsoft, in the paper’s reported results

Limits of the Doctor Comparison

It is not clear from the reported results how MAI-DxO would perform with real patients, full clinical records, physical exams or imaging, or how it would compare with specialists and collaborative care teams. The benchmark’s selected hard cases also do not show how often the system would flag harmless conditions or trigger unnecessary tests in a broader population.

The cost figures are estimates rather than observed hospital bills, and the source report says the final Science paper’s complete methods are not publicly accessible there. The study also does not establish whether using the system changes treatment decisions, patient outcomes, diagnostic delays or total costs. Independent replication has not been reported in the supplied material.

A Prospective Patient Study

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. The source says consenting patients will interact with the system alongside their normal care. Unlike the historical-case benchmark, that setting could help researchers examine how the tool operates during real clinical encounters.

Details such as the study’s size, timing, evaluation measures and safeguards were not provided in the source material. Until results from that work or other independent clinical studies are available, MAI-DxO’s reported scores should be understood as performance on a specific simulated benchmark, not a demonstrated clinical benefit.

Key Questions

What did Microsoft report about MAI-DxO?

In a Science paper published Oct. 22, 2026, Microsoft reported that MAI-DxO scored 83.2% on 304 complex case records and 80.3% on 325 emergency department cases.

Did the study show that AI is better than doctors in hospitals?

No. It compared the system with individual generalist physicians on historical, text-only cases in a simulator. It did not test real patients or compare MAI-DxO with specialists or physician teams.

What did physicians score on the complex cases?

The reported scores were 36.1% with textbooks and internet access and 22.3% without those resources. These figures should be read in light of the study’s selected cases and physician comparison group.

Has the result been independently confirmed?

The paper was peer-reviewed, but the supplied source reports no independent reproduction of the results. Peer review evaluates a paper; it does not by itself establish that other researchers have replicated its findings.

What is expected next?

A prospective study is planned at Beth Israel Deaconess Medical Center in Boston, where consenting patients are expected to interact with MAI-DxO alongside their usual care. The source does not give a timeline or detailed study protocol.

Source: ThorstenMeyerAI.com