What question did this study set out to answer?

This study aims to evaluate the performance of a large language model on clinical reasoning tasks compared to human experts.

May 3, 2026

Performance of a large language model on the reasoning tasks of a physician.

Key Points

This study aims to evaluate the performance of a large language model on clinical reasoning tasks compared to human experts.
Evaluated the LLM against hundreds of physicians across five experiments on complex clinical cases.
Conducted a real-world study in an emergency room comparing AI and human second opinions.
Analyzed performance changes across prior AI generations for clinical decision support.
The LLM outperformed physician baselines in all evaluated experiments.
Notable improvements were seen in the LLM's reasoning capabilities compared to previous AI generations.
The study indicates LLMs have surpassed traditional benchmarks in clinical reasoning.

Abstract

More than 65 years ago, complex clinical diagnostic reasoning cases were introduced as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. In this study, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases across five experiments with a baseline of hundreds of physicians. We then report a real-world study comparing human expert and artificial intelligence (AI) second opinions in randomly selected patients in the emergency room of a major tertiary academic medical center. In all experiments, the LLM outperformed physician baselines and displayed continued improvement from prior generations of AI clinical decision support. Our study suggests that LLMs have eclipsed most benchmarks of clinical reasoning, motivating the urgent need for prospective trials.

Mark Helpful

Bookmark

Relay