Mp17-16 Answering Common Urological Questions- Chatgpt vs. Urology Care Foundation Patient Education Materials

You have accessJournal of UrologyEducation Research I (MP17)1 May 2024MP17-16 ANSWERING COMMON UROLOGICAL QUESTIONS- CHATGPT VS. UROLOGY CARE FOUNDATION PATIENT EDUCATION MATERIALS Adam Schwartz Adam SchwartzAdam Schwartz View All Author Informationhttps://doi.org/10.1097/01.JU.0001008628.15460.84.16AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: ChatGPT is a free online artificial intelligence (AI) chatbot that uses natural language processing to create humanlike responses. Applications of AI natural language processors like ChatGPT in patient education have not yet been thoroughly assessed. This study explores the idea of ChatGPT as a direct source of urological information for patients. METHODS: ChatGPT responses to 30 common urology questions (grouped into 7 topics) were compared to responses to those exact same questions addressed in question/answer format patient education materials (PEM) from Urology Care Foundation (UCF), an established and reliable source of PEM. Posing common urology questions to ChatGPT was intended to simulate how a patient may use ChatGPT directly to learn about a urology topic. The questions used in the study are outlined in Table 1. Flesch-Kincaid Grade Level, Gunning Fog Index, Flesch Reading Ease Score, raw text statistics, the CDC Clear Communication Index tool (CDC CCI), and fact checking were used to evaluate the responses. RESULTS: The average CDC CCI score was 90/100 for UCF responses and 80/100 for ChatGPT responses. ChatGPT scored higher than UCF in 7/30 questions. No false statements were identified. Flesch-Kincaid Grade Level and Gunning Fog Index estimated ChatGPT responses to be written at 12th and 13.9th grade level, respectively (4.9 and 5.3 grade levels higher than UCF responses). A full breakdown of CDC CCI and readability scores is presented in Table 1. Sentences from ChatGPT averaged 53.6% more syllables/sentence (29.2 vs 19) and had 4.9 times 95%CI 3.5, 6.7 greater odds of being over 30 syllables. Words in ChatGPT responses 20% longer (1.8 vs 1.5 syllables/word). CONCLUSIONS: Although clearly outperformed by the UCF responses, ChatGPT still did fairly well on the CDC CCI. With reading grade level estimates ranging from 12-13.9, the biggest issue with ChatGPT as a direct source of patient information seems to be the readability of the responses. Given ½ of Americans can't read beyond an 8th grade level, its likely a significant portion of patients would have trouble understanding at least part of ChatGPT's response. Averaging 53.6% more syllables/sentence than UCF responses, sentence length appears to be a significant shortcoming of ChatGPT as a direct source of urological information for patients. Source of Funding: No funding was used in this study. There are also no conflicts of interest © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e297 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Adam Schwartz More articles by this author Expand All Advertisement PDF downloadLoading ...