Mp24-02 Evaluating the Performance of Different Large Language Models on Health Consultation and Patient Education in Urolithiasis

You have accessJournal of UrologyHealth Services Research: Practice Patterns, Quality of Life and Shared Decision Making II (MP24)1 May 2024MP24-02 EVALUATING THE PERFORMANCE OF DIFFERENT LARGE LANGUAGE MODELS ON HEALTH CONSULTATION AND PATIENT EDUCATION IN UROLITHIASIS Haifeng Song, Yi Xia, Yan Song, Jianxing Li, Guangyuan Zhang, and Bo Xiao Haifeng SongHaifeng Song , Yi XiaYi Xia , Yan SongYan Song , Jianxing LiJianxing Li , Guangyuan ZhangGuangyuan Zhang , and Bo XiaoBo Xiao View All Author Informationhttps://doi.org/10.1097/01.JU.0001008860.46052.c4.02AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: To evaluate the effectiveness of four large language models (LLMs) (Claude, Bard, ChatGPT4, and New Bing) that have large user bases and significant social attention, in the context of medical consultation and patient education in urolithiasis. METHODS: In this study, we developed a questionnaire consisting of 21 questions and 2 clinical scenarios related to urolithiasis. Subsequently, clinical consultations were simulated for each of the four models to assess their responses to the questions. Urolithiasis experts then evaluated the model responses in terms of accuracy, comprehensiveness, ease of understanding, human care, and clinical case analysis ability based on a predesigned 5-point Likert scales. Visualization and statistical analyses were then employed to compare the four models and evaluate their performance. RESULTS: All models yielded satisfying performance, except for Bard, who failed to provide a valid response to Question 13. Claude consistently scored the highest in all dimensions compared with the other three models. ChatGPT4 ranked second in accuracy, with a relatively stable output across multiple tests, but shortcomings were observed in empathy and human caring. Bard exhibited the lowest accuracy and overall performance. Claude and ChatGPT4 both had a high capacity to analyze clinical cases of urolithiasis. Overall, Claude emerged as the best performer in urolithiasis consultations and education. CONCLUSIONS: Claude demonstrated superior performance compared with the other three in urolithiasis consultation and education. This study highlights the remarkable potential of LLMs in medical health consultations and patient education, although professional review, further evaluation, and modifications are still required. Download PPTDownload PPTDownload PPT Source of Funding: None © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e391 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Haifeng Song More articles by this author Yi Xia More articles by this author Yan Song More articles by this author Jianxing Li More articles by this author Guangyuan Zhang More articles by this author Bo Xiao More articles by this author Expand All Advertisement PDF downloadLoading ...