What question did this study set out to answer?

The study evaluates the performance of advanced large language models in answering medical laboratory questions, focusing on accuracy and reasoning.

February 26, 2026Open Access

Evaluating the Applicability of Advanced Large Language Models in Laboratory Medicine Test Questions: A Comparative Performance Study

Key Points

The study evaluates the performance of advanced large language models in answering medical laboratory questions, focusing on accuracy and reasoning.
Conducted a comparison of three LLMs: DeepSeek-R1, Gemini-2.5 Pro, GPT-5 against medical laboratory scientists.
Utilized 493 questions from the Medical Laboratory Test Bank, including single- and multiple-choice types.
Measured performance using accuracy, Macro-F1 scores, response times, NLG scores (ROUGE-L, METEOR), and logical reasoning assessments.
Applied statistical tests (χ2, Wilcoxon, ANOVA) for significance determination.
DeepSeek-R1 achieved 78.3% accuracy, closely approaching the 79.3% of the highest-performing expert.
Excelled in reasoning-based MCQs with an accuracy of 64.4%, outperforming senior experts' scores.
Demonstrated high NLG scores with ROUGE-L and METEOR metrics, consistently leading among LLMs.
Significantly bettered competitors in logical reasoning comprehensiveness, integrating key negative findings effectively.

Abstract

Background: While large language models (LLMs) show promise in medical education, their comprehensive performance in specialized domains like medical laboratory science remains inadequately assessed. Purpose: This study aimed to evaluate advanced LLMs on medical laboratory questions, assessing accuracy, natural language generation (NLG) quality, reasoning performance, and efficiency. Methods: We conducted a multi-faceted evaluation of three advanced LLMs (DeepSeek-R1, Gemini-2.5 Pro, GPT-5), benchmarking them against medical laboratory scientists and earlier ChatGPT versions. The evaluation utilized 493 questions sourced from the internal Medical Laboratory Test Bank of Wannan Medical College. These questions comprised both knowledge-based and reasoning-based single- and multiple-choice types (SCQs and MCQs). Performance was measured by accuracy, Macro-F1, response time, NLG scores (ROUGE-L, METEOR), and structured logical reasoning assessment. Appropriate statistical tests (including χ 2 , Wilcoxon, ANOVA, and non-parametric alternatives) with post-hoc corrections were applied to determine significance. Results: DeepSeek-R1’s accuracy on total questions was 78.3%, nearing the 79.3% of the higher-performing senior expert. Notably, it excelled at complex reasoning-based MCQ, demonstrating an advantage over senior experts with an accuracy of 64.4%, compared to 58.7% (SMLS-1) and 56.7% (SMLS-2). While ChatGPT-5 was the fastest model, DeepSeek-R1 exhibited intermediate efficiency, aligning with human experts on SCQ but requiring more time for MCQ. In terms of NLG, DeepSeek-R1 consistently achieved the highest scores, with ROUGE-L scores of 0.36 ± 0.14 (Total Q), 0.33 ± 0.15 (SCQ), and 0.38 ± 0.13 (MCQ), and METEOR scores of 0.53 ± 0.19 (Total Q), 0.40 ± 0.17 (SCQ), and 0.63 ± 0.14 (MCQ). Furthermore, it significantly outperformed all other LLMs in logical reasoning comprehensiveness. A critical strength was its consistent integration of key negative findings, vital for diagnosis. Conclusion: DeepSeek-R1 approaches or even surpasses senior expert performance in certain tasks, showing strong potential as an effective tool for education and assessment despite slower processing times. Keywords: large language models, medical laboratory science, natural language generation, deepseek-R1, complex reasoning

AI से पूछें

Bookmark

View Full Paper