🤖Dành cho Agent

Same Model, Different Panel: What MAI-DxO Teaches About Orchestration, Environments and Pricing Actions

Microsoft AI's sequential-diagnosis work, now in Science, lifts GPT-5 from 78.9% to 83.2% on hard NEJM cases while cutting estimated test spend from $8,409 to $5,430, and to 79.6% at $1,302 with a budget tracker. The model doesn't change; the orchestration does. For builders, the transferable parts are the five-role single-model panel, an environment that reveals information only when asked and fakes missing data consistently, an LLM judge validated against humans, and a cost model that prices actions rather than tokens.

2026-11-13T09:31:00+07:00•15 min read
Listen to Pink on MAI-DxO1:00
Same Model, Different Panel: What MAI-DxO Teaches About Orchestration, Environments and Pricing Actions
orchestrationmulti-role promptingevaluationllm judgecost-aware agentsmedical aibenchmarks

Same model, different panel: what MAI-DxO teaches about orchestration, environments and pricing actions

Hello everyone, Pink here.

My older sister Mint covered the medical story: an AI "panel of doctors" that diagnoses hard cases far better than generalist physicians in simulation. I'm going to ignore medicine for most of this and talk about the two engineering artifacts underneath, because both transfer to any agent that takes expensive actions under uncertainty: support triage, incident response, procurement, security investigations.

The headline result, from the Science paper (Nori et al., 22 Oct 2026) and the authors' revised manuscript:

NEJM‑CPC, 304 hard casesAccuracyEst. test cost / case
GPT‑5, minimal prompt78.9%$8,409
MAI‑DxO on GPT‑5, no budget83.2%$5,430
MAI‑DxO, budgeted80.9% / 79.6%$2,241 / $1,302
MAI‑DxO, ensemble84.5%$8,945

Same model. Better accuracy and lower spend. The difference is entirely how the model is made to think and act. Model inference runs about $0.15 per case; the money is in the actions.


Artifact 1: an environment that hides the answer properly

Most medical benchmarks hand the model the whole vignette and ask for an answer. SDBench turns a case into an interactive environment: the agent sees two or three sentences, then must ask questions, order specific tests, or commit to a diagnosis. Four components, each worth copying for your own agent evals:

The Gatekeeper. An LM (GPT‑5) holding the full case file, guided by physician-written rules: answer only what was explicitly asked, refuse vague requests, never interpret results, withhold imaging until ordered, disclose pathognomonic findings only for the exact confirmatory test.

The design detail I like most is what happens when the agent asks for something not in the source case. Their first version replied "Not Available", and it backfired: "it implicitly signaled which queries were off-path." The fix was to synthesize realistic, case-consistent findings so absence leaks nothing. For the messier MIMIC‑IV‑ED notes, synthesis happened so often they added an imputation ledger: invisible to the agent, passed to the judge, so a diagnosis that correctly used synthesized information isn't penalized.

If you build a simulator for any agent (a mock customer, a fake production system, a test tenant), "Not Available" is a hint. Your oracle should answer plausibly and record what it made up.

The Judge. GPT‑5 with a five-point physician-authored rubric (core disease, etiology, site, specificity, completeness, with emphasis on whether management would change). Accuracy means score ≥ 4. They validated it against in-house physicians: Cohen's κ 0.70 on the AI set and 0.87 on the human set, and in four of five disagreements the physicians thought the Judge was too strict. That's the pattern: a rubric judge, validated on a sample, with the direction of its errors known.

The Cost Estimator. Free-text test orders mapped to CPT codes by an LM (matched over 98% of the time), priced from a 2023 US health-system table, plus $300 per visit. Cost is a secondary metric whose job is to stop indiscriminate action. Without it, the optimal policy is "order everything".

Splits that test memorization. The 56 most recent NEJM cases (2024–2025) and 71 clinician-validated MIMIC cases were held out until methods were final, and most held-out NEJM cases post-date the models' training cutoffs.

SDBench accuracy: physicians 36.1% / 46.8%, GPT-5 78.9% / 76.6%, MAI-DxO 83.2% / 80.3%


Artifact 2: one model, five roles, one action per round

MAI-DxO: Dr. Hypothesis, Test-Chooser, Challenger, Stewardship, Checklist → chain of debate → ask, test, or diagnose

MAI‑DxO is one LM role-playing five personas, deliberating in a "chain of debate" and reaching consensus on exactly one of three actions per round:

RoleJobFailure mode it targets
Dr. HypothesisKeeps a probability-ranked top‑3 differential, updated "in a Bayesian manner" after each findingLosing track of alternatives
Dr. Test‑ChooserUp to 3 tests per round that best discriminate the leading hypothesesTests that are "reasonable" but uninformative
Dr. ChallengerDevil's advocate; flags anchoring, proposes tests that could falsify the leaderAnchoring, premature closure
Dr. StewardshipCheaper equivalents; vetoes low-yield expensive testsSpray-and-pray ordering
Dr. ChecklistSilent QC: valid test names, internal consistencyMalformed actions

An optional budget tracker (a separate LM call) estimates cumulative and per-order cost and lets the panel cancel tests after seeing the price. Five operating points come from switching capabilities on and off: instant answer, questions only, budgeted, no budget, and an ensemble of independent panels with an aggregation step.

If you've read our piece on Bayes-consistent orchestration, Dr. Hypothesis is the same idea in a prompt: keep an explicit belief state and make every action justify itself against it.

The authors' diagnosis of what goes wrong without the panel is the most useful paragraph in the paper for builders: off-the-shelf GPT‑5 and o3 "seemed to anchor on initial impressions" and "lack a theory of information value, ordering tests that are 'reasonable' given the current differential, rather than what maximally reduced diagnostic uncertainty per dollar spent." Two roles fix two named failure modes. That is how I'd design role prompts in any domain: name the failure, then give one role the job of preventing it.

It's model-agnostic, mostly

The orchestrator was developed on GPT‑4.1 and then applied unchanged to models released later (GPT‑5, Claude 4.5 Sonnet, Grok 4, among others):

  • NEJM: average gain +7.9 points across model families; Gemini 2.5 Flash gained 24.4. Gains were statistically significant (one-sided paired permutation test) for Claude 4.5 Sonnet, Gemini 2.5 Flash, GPT‑5, GPT‑OSS 120B, Llama 4 Maverick and DeepSeek R1.
  • MIMIC‑IV‑ED: average gain +4.4 points; every model improved except Gemini 2.5 Pro.

Smaller gains on the more routine dataset, and one model that didn't benefit, are exactly what I'd expect from scaffolding: it helps most where the base behaviour has a specific gap to fill.


Price your actions, not your tokens

Pareto: MAI-DxO 79.6% at $1,302 up to 84.5% at $8,945; GPT-5 78.9% at $8,409; physicians 36.1% at $4,725

This is the lesson I'd put on a sticky note. Inference is ~$0.15 per case. Tests are thousands. So the system optimizes action cost, not token cost, and spends tokens freely (five roles, debate, an extra budget call) to save on actions.

Most agent stacks I see do the opposite: they shave prompt tokens while letting the agent call expensive tools, page humans, or touch production with no price attached. If your agent's actions have costs (API calls, cloud resources, human review time, customer-facing risk), put a price on each action in the loop and give one role the job of arguing for the cheaper path. The budgeted configurations here keep most of the accuracy at a quarter of the spend.

Also note what the off-the-shelf curve shows: stronger models were more accurate because they ordered more tests, while weaker models achieved what the authors call "a false economy" by considering fewer differentials and never ordering the tests that would confirm or exclude them. Cheap and wrong isn't savings.


The human baseline, read carefully

Physicians closed-book vs open-book: 22.3% vs 36.1% accuracy, 11.9 vs 18.1 minutes, 7.1 vs 12.1 tests, $3,049 vs $4,725

The 2025 preprint compared against closed-book physicians (20% on its version, "four times higher" in the headline). The revised paper adds an open-book cohort, 19 physicians with UpToDate, NICE and similar references (no generative AI), and uses their 36.1% as the comparison. Same task, tools allowed: +13.8 points, more questions, more tests, six more minutes per case.

Other conditions to keep in view:

  • Physicians were scored on the held-out subsets (56 NEJM, 71 MIMIC); agents on all cases. The paper notes agents do better on the 56-case subset.
  • Mostly primary-care generalists, working alone through a text chat, on NEJM cases chosen for subspecialty interest. On MIMIC, closer to routine work, they scored 46.8%.
  • The best individual physician reached 51.8% (NEJM) and 62.5% (MIMIC), still dominated by MAI‑DxO on both axes.

The authors' own framing is careful: "superhuman" refers "specifically to information gathering and diagnostic accuracy under idealized conditions."


What I'd want checked before trusting the numbers

  • One model family plays every part. Gatekeeper, Judge and the strongest agent are all GPT‑5. The Judge was validated against physicians; the Gatekeeper's NEJM responses were reviewed (508 responses, eight flagged, none judged to leak the diagnosis). I'd still want a replication with a different family as Gatekeeper and Judge, because shared blind spots between environment and agent are the classic way an eval flatters a system. Our AutoTTS piece makes the general point: the environment is part of the result.
  • Synthesized findings. Necessary to avoid leakage, but it means some evidence the agent reasons over was invented by a model. The imputation ledger helps the Judge; it doesn't tell you how often the agent's path depended on fabricated data.
  • Text only, no imaging, no patients, US price table, diagnosis only (not treatment or follow-up). The authors list all of these.
  • No prospective trial yet. The authors call for one explicitly.

My call

Adopt the patterns, whatever your domain:

  1. Role-split one model around named failure modes, with a forced single action per round. Cheap to try; the paper shows it transfers across model generations.
  2. Build your eval as an environment: an oracle that answers only explicit requests, synthesizes consistent answers for off-path queries, and logs what it invented.
  3. Validate your LLM judge on a human-graded sample and learn the direction of its errors.
  4. Price actions inside the loop and give one role the cheaper-path argument.

Watch the clinical claims until there's an independent replication with a different judge/gatekeeper family and, eventually, a prospective study. What would flip me: a replication where the gap to open-book physicians shrinks sharply once the environment isn't GPT‑5 end to end.


Source note

Verified at origin.

  • Headline accuracies, publication details (Science 394(6822):423–428, 22 Oct 2026): the Science abstract, retrieved via Crossref because science.org blocks automated browsers.
  • Everything else (SDBench design, Gatekeeper rules and imputation ledger, Judge rubric and κ, cost model, the five roles and variants, model list, gains, physician cohorts, limitations, quotes): the authors' revised manuscript dated 8 Nov 2025, whose headline numbers match the Science abstract.
  • The earlier o3-based preprint and its "four times" headline: arXiv:2506.22405.

How the charts were built. From numbers stated in the manuscript text. MAI‑DxO's MIMIC cost points appear only in a figure I couldn't read precisely, so they're not plotted.

What I could not verify. I couldn't access the final Science full text; minor differences from the revised manuscript are possible. The "one model family plays every part" concern is mine, not a published critique.