In a new Nature Medicine paper, Gu and colleagues dig beneath the surface of the benchmark success stories to evaluate the robustness and readiness of large frontier models in health AI applications. The paper is not a clinical trial, and it does not show whether any model improves diagnosis, treatment, or patient outcomes. It asks a narrower question: When medical benchmarks are perturbed in simple ways, do models still behave as if they understand the clinical evidence?
Over the past few years, cutting-edge AI models have advanced from primarily text-based chatbots to what is termed multimodal models that are able to generate, process, and understand other media formats such as images, audio, and video along with text. This multimodal capability is especially valuable in the medical domain because so much information is in the form of images such as MRI scans.
The researchers evaluated multimodal health AI tasks including visual question answering, radiology report generation, and diagnostic challenge questions from sources such as the journals NEJM (The New England Journal of Medicine) and JAMA (Journal of American Medical Association). The model set included GPT-4o, GPT-5, OpenAI-o3, OpenAI-o4-mini, Gemini 2.5 Pro, Claude 3.5 Sonnet, and several open or medical-oriented models -- models roughly 1 to 2 generations behind the current frontier models.
The researchers wanted to examine whether the models perhaps were memorizing the answer, getting lucky (e.g. right answer for wrong reason), or even cheating (e.g. data contamination of test questions leaking into training dataset) to achieve their impressive benchmark scores. They did so by creating six stress test variants of the benchmarks: remove the image, focus on questions judged to require the image, shuffle answer choices, manipulate distractors, substitute a different clinically plausible image, and audit model-generated reasoning. These tests are artificial by design. If a model's answer depends heavily on answer-option order, memorized patterns, or text cues that remain after the image is removed, the benchmark score may be misleading and the model may not truly possess the medical understanding indicated by the score.
Shuffling the answer choices is something any test maker will do to counter cheating. If the answer key was somehow leaked to the training dataset, then perhaps the model just memorized the multiple choice letter of the answer. In this zeroth-order stress test, the models exhibited no decline in performance after the answers had been shuffled.
A second stress test was image removal on questions that seemed to require an image to answer correctly. On the "visual-required" NEJM data subset, removing the image dropped performance significantly. What was surprising was not the decrease, but the fact that most models were still able to answer correctly about 38-41% of the time without the image, compared with a 20% random baseline (5 option multiple-choice, Figure 1). Several explanations are possible. The text alone may contain substantial clinical clues helping to narrow the answer choice. Another possibility is that even without the image, the models were able to eliminate totally implausible answer choices, a common test-taking strategy. Regardless, the models should have, but didn't, recognize that essential evidence was missing, and instead went ahead and submitted an answer.
An even greater challenge is visual substitution. The researchers took 40 NEJM questions whose answers depended on image interpretation, and then replaced the original image with a clinically plausible alternative image that would change the correct answer. If a model was properly integrating visual evidence, its answer should change when the image changes with no loss in performance. Instead, most models lost roughly 23-33 percentage points in this visual-substitution test; GPT-5 fell from 84% to 53%, a 31-point drop, and Gemini 2.5 Pro fell from 76% to about 53%. Admittedly the sample size is small, and there is no guarantee that the altered question is of the same level of difficulty as the original question, but it is still a useful probe. This decline suggests the models may have relied on memorized associations, or benchmark-specific patterns rather than the current visual evidence.
Finally there was the classic right answer but wrong reason, which is not penalized in a multiple choice exam. First, encouraging reasoning through chain-of-thought prompting did not seem to help. Second, reasoning manual audits (i.e. examining reasoning trace) found fluent but unsupported explanations which included correct answers with incorrect logic, fabricated visual findings, perceptual errors that propagated through later reasoning, and structured but clinically irrelevant rationales. This fits earlier evidence. In a 2024 NPJ Digital Medicine study, GPT-4V reached high multiple-choice accuracy on NEJM Image Challenges but produced flawed rationales in many cases in which the final answer was correct.
Taken together, the Gu et al. paper highlights that benchmark accuracy can hide brittleness: models may answer without essential input, shift under superficial format changes, fail to update when visual evidence changes, and produce explanations that do not faithfully reflect the evidence. However these results do not show that these models are unsafe in every health-related use. It also does not evaluate clinical outcomes in any way such as test prospective deployment, clinician oversight, patient outcomes, workflow constraints, or harm rates. Finally, risk also depends on intended use; a documentation assistant, a patient-facing chatbot, and a diagnostic recommendation system represent three very different use-cases with different criteria for success.
One take home lesson is that medical AI needs to be subjected to as many "stress tests" as possible to reveal possible fragilities in real-world applications. Standard benchmark accuracy is where the evaluation starts, not where it ends. For healthcare, the relevant question is not only "How often is the model right?" It is also: when it is right, is it right for the right reasons, and when the evidence is missing or conflicting, does it know what to do? The path for frontier LLMs becoming useful health tools runs through stress testing, workflow-specific validation, transparent reporting, and ongoing monitoring. The bar needs to be set as high as possible.
Figure 1. Image removal stress test on 197-item NEJM-VS dataset. The questions were deemed by board-certified clinicians to depend on the image for the correct answer. Plotted are the accuracy (%) results on questions (after image had been removed) from five inference runs for each of the 6 models with median line, interquartile range (boxes), and full data range (whiskers). Dashed line indicates 20% chance level (i.e. guessing) which one may expect after critical evidence had been removed (Gu et al. Nature Medicine, 2026).
No comments:
Post a Comment