General-purpose large language models outperform specialized clinical AI tools on medical benchmarks

“A new rigorous study compared the generalized AI frontier models (OpenAI, Anthropic, and Google) vs OpenEvidence and UpToDate AI specialized models. Surprisingly, the generalized models substantially outperformed the latter for medical information”

“Clinical AI tools may carry institutional legitimacy and are likely safe for routine use, but our results show that they are not superior to frontier models on knowledge, communication or clinical alignment”

“Interesting and important study. However, specialized tools such as OpenEvidence and UpToDate Expert AI were largely developed to provide rapid access to curated evidence, whereas frontier LLMs are increasingly optimized for reasoning and synthesis. Both capabilities are valuable, but they are not necessarily the same. More importantly, this evaluation was conducted in a relatively static environment. Real-world clinical practice is inherently dynamic, involving evolving patient conditions, iterative decision-making, incomplete information, and frequent uncertainty. Demonstrating superiority in static benchmarks is important, but whether this translates into superior performance in the dynamic reality of clinical care remains an open and highly relevant question.”