B-147 Evaluation of the specialized Medicine-GPT for adolescent health inquiries related to diagnostics and laboratory tests

Background Large language models (LLMs) are generative AI systems designed to simulate human-like conversation. Recently, LLMs have advanced in understanding questions, retrieving accurate answers, and performing sophisticated reasoning—capabilities that improve access to medical information. In particular, specialized medical LLMs have been developed by fine-tuning general LLMs on domain-specific medical datasets. However, performance evaluations of medical LLMs in the field of Pathology and Laboratory Medicine remain limited, and it is unclear how well these fine-tuned models manage diagnostic and laboratory queries. Adolescents, who are typically more open to new technologies, increasingly rely on LLMs for health-related information. This study compares Medicine-GPT with its predecessor, GPT-4, to evaluate their ability to provide accurate, comprehensive, and effective responses to real-life questions about laboratory results and diagnostics posted by adolescents. Methods We gathered over 100 clinical questions and scenarios from Reddit’s “Ask Doctors” subreddit, filtering for posts authored by adolescents and sorted by “Top” interactions. Only questions related to lab medicine and diagnostics were selected. A total of 30 questions were processed using two AI models: ChatGPT-4 (free version) and Medicine-GPT. A panel of medical professionals evaluated the responses based on five criteria: factual correctness, completeness, clarity, reasoning, and helpfulness. Each criterion was scored numerically—1 for the highest performance, 2 for partially satisfactory, and 3 for the lowest. Differences between the groups were analyzed using the chi square test. Results Thirty questions were analyzed, covering diverse topics such as endocrinology (14 questions on issues including puberty, fertility, diabetes, and hypertension), hematology (4), general chemistry (3), microbiology (3), molecular pathology (2), immunology (2), and toxicology (2). Medicine-GPT achieved a perfect score across all categories in 13 responses, compared to only 1 perfect response by ChatGPT-4. Both models demonstrated complete factual accuracy with no hallucinations. Significant differences were observed in completeness, reasoning, and helpfulness scores: Medicine-GPT achieved ratings of 66.6% at level 1 for completeness (versus 20% by ChatGPT-4), 60% at level 1 for reasoning (versus 33.3%), and 46.6% at level 1 for helpfulness (versus 23.3%) (p-values = 0.0001, 0.0384, and 0.0220, respectively). Both models performed similarly in clarity, with Medicine-GPT receiving 80% at level 1 compared to 70% by ChatGPT-4. A common issue with Medicine GPT and GPT-4 is that they present numerous possibilities - both common and rare, to the users, which can leave teenagers feeling very confused or even overwhelmed, especially when rare and fatal conditions are presented as possibilities. It is important to note that the answers LLMs provide may differ from a doctor*s usual work-up – starting with the most likely cause and then moving to the less likely ones. Conclusion Medicine-GPT shows promise as a powerful tool for addressing adolescent health inquiries, outperforming ChatGPT-4 in completeness, reasoning, and overall medical helpfulness. However, its tendency to present excessive diagnostic options may confuse young users, indicating a need for improvements in guiding the interpretation of multiple possibilities and enhancing critical reasoning in response presentation.