# Did Microsoft AI Really Beat Doctors At Diagnosis? What The Study Actually Tested

Source: https://in-store.ca/blog/sequential-diagnosis-benchmark-reality
Published: 2026-10-31

Reviewed by: Andrew Liashok, Client Service Director, IN-STORE (https://in-store.ca/team/meet-andrew)
Publisher: IN-STORE, Toronto, Canada

> What Microsoft’s MAI-DxO and SDBench actually measured, the verified accuracy numbers, and why simulated “cost” differs from real world savings.

## What MAI-DxO Actually Measured, And Why The Viral 83% Vs 36% Framing Misleads

“Microsoft AI beat doctors at complex diagnosis” makes for a clean headline. The underlying work is messier, and far more useful for leaders who care about how AI changes decision making in high stakes settings.

What Microsoft describes is a benchmark for sequential diagnosis: a stepwise process where a system asks follow up questions, orders tests, and only then commits a diagnosis. The big lesson for businesses in Canada and the United States is less about bragging rights, and more about a new pattern: AI as a process coordinator that can trade off accuracy and cost, under rules you define, with audit points you can inspect.

## What The Research Actually Supports

Microsoft’s own materials describe a system called the Microsoft AI Diagnostic Orchestrator (MAI-DxO) evaluated on sequential diagnosis tasks built from New England Journal of Medicine Clinical Pathological Conference cases. In Microsoft’s reporting, MAI-DxO paired with OpenAI’s o3 “correctly solved 85.5% of the NEJM benchmark cases,” while “21 practicing physicians” achieved a “mean accuracy of 20% across completed cases.”

In the associated arXiv preprint, the authors describe converting “304 diagnostically challenging” NEJM CPC cases into stepwise encounters, then scoring performance on diagnostic accuracy and the cost of visits and tests. The abstract reports “80% diagnostic accuracy” for MAI-DxO paired with OpenAI’s o3, describes that as “four times higher than the 20% average of generalist physicians,” and states that “When configured for maximum accuracy, MAI-DxO achieves 85.5% accuracy.”

## What “Sequential Diagnosis” Means In This Benchmark

### The Point Of SDBench

Microsoft’s SDBench describes sequential diagnosis as a turn by turn workflow. Instead of answering a one shot multiple choice question, the diagnostic system starts with a brief, non revealing case summary and must iterate: ask questions, order tests, or commit a diagnosis “just as a clinician works up a real case.”

SDBench is scored on two outputs: diagnostic accuracy and the cost of the tests ordered. Practically, that framing pushes any system into an accuracy cost trade off, which mirrors what businesses face when they use AI for any multi step decision process.

### The Agent, Gatekeeper, Judge Loop

SDBench formalizes roles. The “Diagnostic Agent” receives a short summary and decides, turn by turn, whether to order tests, ask questions, or commit a diagnosis. A “Gatekeeper” answers each test order or question using the case’s full text, returning the real result when present, or a realistic synthetic result when the case does not contain it. A “Judge” then scores the committed diagnosis against ground truth on a 1 to 5 Likert scale, where 4 to 5 counts as correct.

That structure matters because it tells you what kind of performance is being measured: skill at driving a constrained information gathering loop, followed by an LLM graded rubric score. It also reveals what your organization would need to replicate for any serious deployment in a different domain: controlled information access, standardized outputs, and a defined scoring rule.

### Which Cases Were Used

Microsoft describes the NEJM cases as “Case Records of the Massachusetts General Hospital” and frames them as “among the most diagnostically complex.” The Microsoft blog says SD Bench “transforms 304 recent NEJM cases into stepwise diagnostic encounters,” matching the arXiv description of “304” NEJM CPC cases transformed into stepwise encounters.

SDBench also describes licensing constraints. The NEJM based dataset must be licensed from NEJM, and access to the MIMIC dataset requires PhysioNet credentialing, training, and acceptance of the data use agreement. SDBench includes a small “Synthetic” dataset of five fully synthetic NEJM style cases for open evaluation, and its documentation warns this synthetic set should not be used to report diagnostic performance.

## How MAI-DxO Works, According To Microsoft

Microsoft describes MAI-DxO as “a system designed to emulate a virtual panel of physicians with diverse diagnostic approaches collaborating to solve diagnostic cases.” The point is orchestration. It can ask follow up questions, order tests, or deliver a diagnosis, then run a cost check and verify its own reasoning before deciding whether to proceed.

For business leaders, the important idea is the orchestrator pattern: a layer that turns a general language model into a governed process that can pause, request more information, and enforce a budget. Many organizations are trying to do this informally with prompts and ad hoc checklists. SDBench shows what it looks like when the loop is formalized, scored, and tuned.

## The Cost Claims, And Why They Do Not Transfer Cleanly To Real Life

The arXiv preprint says performance is assessed using diagnostic accuracy and the cost of physician visits and tests performed. Microsoft’s blog similarly describes that each requested investigation incurs a “(virtual) cost,” enabling evaluation of “diagnostic accuracy and resource expenditure.”

The costing method matters. The sdbench GitHub documentation says the repository supports LLM based estimates of diagnostic test costs and estimates grounded on a public hospital charge master. It also states that in the published paper, costs were grounded against a public hospital charge master, and warns that LLM cost estimates are not directly comparable to grounded cost estimates. The README reports that the “gpt-5-mini” estimates were “40-50% higher” than the grounded estimates used in the paper.

Microsoft also flags a real world constraint in plain language: health costs vary across geographies and systems and include downstream factors that are not accounted for, even when applying a consistent methodology. So “reduces diagnostic costs by 20% compared to physicians” (from the arXiv abstract) is best read as a benchmark output under a defined costing scheme, not a transferable promise of savings inside a real clinic, insurer network, or employer benefits plan.

## Key Limitations That Change The Interpretation

### The Physician Baseline Was Constrained

Microsoft says 21 practicing physicians took part, and also states clinicians worked without access to colleagues, textbooks, or generative AI, to enable a comparison to raw human performance. Whatever you think about that design choice, it means the headline “AI beat doctors” does not describe a typical clinical workflow, which often includes consultation, references, and team review.

### This Was A Research Demonstration

Microsoft’s blog states that SD Bench and MAI-DxO are research demonstrations only. It also states that while MAI-DxO excels at complex diagnostic challenges, further testing is needed to assess performance on more common presentations.

### The Benchmark Confirms A Type Of Capability

SDBench evaluates a system’s ability to drive an iterative workup under constrained information access and then commit to an answer that is graded with a rubric. That is meaningful. It also means the benchmark is strongest as evidence of progress in agentic, stepwise decision processes, rather than proof of safe clinical deployment.

## Why Regulation Becomes The Business Problem

For Canadian operators, Health Canada’s “Pre-market guidance for machine learning-enabled medical devices” (Publication date: February 2025) describes machine learning enabled medical devices (MLMD) as devices using ML for an intended medical purpose, and states they are subject to the Food and Drugs Act and associated Medical Devices Regulations. The guidance is aimed at manufacturers submitting a new or amendment application for Class II, III and IV MLMD, and it introduces concepts such as a predetermined change control plan (PCCP) and terms and conditions (T&Cs) on medical device licences.

In the United States, the FDA’s “Guidances with Digital Health Content” index lists guidance documents including “Clinical Decision Support Software” (Issue Date 01/29/2026, Final), “Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions” (Issue Date 08/18/2025, Final), and “Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations” (Issue Date 01/07/2025, Draft).

Even for leaders outside healthcare, this is the signal: once AI moves from content and analysis into decisions that carry safety and liability, the work shifts from “can it answer” to “can we validate, control, and change manage it.” The orchestrator pattern makes that shift more concrete because it yields checkpoints: questions asked, tests ordered, budget applied, and verification steps run. Those checkpoints are where governance can attach.

## What SMB Leaders Should Take From This

If you run a service business, you can treat this as a case study in supervised automation for multi step work. The transferable idea is sequential execution with explicit rules: make the system ask for missing inputs, force it to choose next steps from a constrained menu, score the output against a rubric, and record the path taken.

- **Define the loop:** what questions must be asked before a recommendation can be made, and what information is allowed to be used at each step.
- **Separate accuracy from cost:** decide what “cost” means in your context (time, paid tools, staff touches), then test accuracy cost trade offs deliberately.
- **Design checkpoints for review:** MAI-DxO’s described cost check and self verification step is a reminder to add explicit review points before output goes to a customer.

This is where an AI-assisted growth agency can add real value. Many teams adopt automation as a set of disconnected tools. The more durable approach looks like SDBench: a managed process, measured outputs, and a documented record of what happened turn by turn.

Microsoft’s reported results around MAI-DxO and SDBench support a clear claim: sequential, stepwise diagnostic reasoning in a simulated benchmark has improved sharply, and the measured outputs can look dramatic. The more durable lesson for business leaders is the orchestrator pattern: governed, logged processes that can balance quality and cost.

If you want help turning that pattern into marketing operations (intake, content planning, local visibility, and lead workflows) with human review built in, IN-STORE can help as a [monthly marketing partner](https://www.in-store.ca/).

Sources:

- [https://drive.google.com/file/d/1ZwcbAcHWMQU9XY12QJLrMQQFitMRXwZS/view?usp=drivesdk](https://drive.google.com/file/d/1ZwcbAcHWMQU9XY12QJLrMQQFitMRXwZS/view?usp=drivesdk)
- [The Path to Medical Superintelligence](https://microsoft.ai/news/the-path-to-medical-superintelligence/)
- [Sequential Diagnosis with Language Models (arXiv:2506.22405)](https://arxiv.org/abs/2506.22405)
- [SDBench , Sequential Diagnosis Benchmark](https://microsoft.github.io/sdbench/)
- [Pre-market guidance for machine learning-enabled medical devices](https://publications.gc.ca/collections/collection_2025/sc-hc/H164-354-2025-eng.pdf)
- [Guidances with Digital Health Content | FDA](https://www.fda.gov/medical-devices/digital-health-center-excellence/guidances-digital-health-content)