Microsoft MAI-DxO: New SDBench Results, Open Code and Clinical Limits

Microsoft published new MAI-DxO diagnostic-AI results on October 22 alongside SDBench, its sequential diagnosis benchmark. The research evaluates how systems request information and choose tests before reaching a diagnosis. It is a research result and evaluation release, not an announcement that an autonomous doctor is ready for routine care.

What Microsoft reported

In its announcement of the Science study, Microsoft reports the following diagnostic accuracies under the study's evaluation conditions:

Case set	MAI-DxO	Physician comparison
Complex NEJM cases	83%	36% with external references; 22% without
More typical MIMIC-IV-ED emergency cases	80%	47%

These are Microsoft-reported benchmark results, not measured improvements in patient outcomes or a promise about a hospital's bill.

MAI-DxO coordinates model roles that propose diagnoses, challenge them and select questions or tests. This follows earlier research, rather than introducing an entirely new evaluation approach.

SDBench tests a sequence, not one answer

The SDBench project page explains the three roles in its evaluation loop. The diagnostic agent begins with a limited case summary. A gatekeeper provides information when the agent asks questions or orders tests. A judge then scores the final diagnosis against the reference answer.

An important detail is that the gatekeeper can generate a synthetic result when the original case does not contain the requested information. The judge uses a five-point scale, with four or five counting as correct.

That makes the benchmark more interactive than a multiple-choice exam, but it also introduces another layer to inspect. A score depends on the agent, the information the gatekeeper supplies and the way the judge evaluates the answer. Treating it as a direct observation of clinical practice would erase those distinctions.

For AI builders, the broader technical idea is useful beyond medicine. An agent's final answer does not show whether it asked an unnecessary question, ordered an expensive operation or stopped too early. Evaluating the sequence makes those choices visible.

Open code does not mean unrestricted clinical data

The project separates its code from the case data. NEJM data requires licensing. MIMIC access requires PhysioNet credentialing, training and acceptance of its data-use agreement. Five fully synthetic cases are included for access-free development, explicitly not for reporting diagnostic performance.

This matters when someone describes a quick reproduction. A successful run on the bundled examples can establish that the harness works. It cannot establish that a new agent matches the published medical results.

The repository documents a science configuration that fixes the gatekeeper and judge for each dataset. Published comparisons use the combined validation and test sets. Those choices need to remain visible when comparing a different diagnostic agent.

The cost comparison has its own configuration

There is a second reproduction issue in the repository: the default cost calculation uses model-generated estimates, while the paper used estimates grounded in a public hospital charge master.

The maintainers report that their default GPT-5-mini estimates were 40% to 50% higher than the grounded estimates used in the paper. They also say those two methods are not directly comparable. Grounded costing can still fall back to a model estimate when a test cannot be matched.

Accuracy and estimated cost need separate audit trails. Record the dataset, agent, judge, gatekeeper and costing method together. Otherwise, a lower number could reflect a different accounting method rather than a better decision process.

This is particularly important for agent research because the cheapest next action is not always the most useful one. The measurement should show whether the system achieved the task with less unnecessary work, rather than merely selecting actions assigned lower prices.

What remains to be established in practice

Microsoft says a new partnership with Beth Israel Deaconess Medical Center will evaluate clinical reasoning across hospital settings, including a prospective study with direct patient interaction and independent physician assessments. That work is prospective, not a completed real-world validation announced here.

The evidence still needs to connect benchmark performance with how a system behaves when information is incomplete, workflows are interrupted and clinicians must act on the result. This article describes the research and developer tooling, not medical advice or a recommendation to use it for personal diagnosis.

The next meaningful result will be whether better sequential reasoning survives the move from a controlled evaluation into care, with the limitations measured as carefully as the improvements.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams explain complex AI research and enterprise products clearly, without turning evaluation results into promises the evidence cannot support.

Book a 30-minute call.