PD28-02 ASSESSING THE QUALITY AND PUBLIC PERCEPTION OF CHATBOT-GENERATED VS. PHYSICIAN ANSWERS TO SENSITIVE MEN's HEALTH CONCERNS

You have accessJournal of UrologySexual Function/Dysfunction: Evaluation I (PD28)1 May 2024PD28-02 ASSESSING THE QUALITY AND PUBLIC PERCEPTION OF CHATBOT-GENERATED VS. PHYSICIAN ANSWERS TO SENSITIVE MEN's HEALTH CONCERNS Jacob S. Hershenhouse, Daniel Mokhtar, Lorenzo Storino Ramacciotti, Severin Rodler, Brian Hom, Josh Ku, John Tran, Andre Abreu, Giorgio Ivan Russo, Ege Can Serefoglu, Andrea Cocci, Kian Asanad, Andrey Morozov, Ioannis Sokolakis, Afonso Morgado, Fabio Castiglione, Andrea Salonia, and Giovanni E. Cacciamani Jacob S. HershenhouseJacob S. Hershenhouse , Daniel MokhtarDaniel Mokhtar , Lorenzo Storino RamacciottiLorenzo Storino Ramacciotti , Severin RodlerSeverin Rodler , Brian HomBrian Hom , Josh KuJosh Ku , John TranJohn Tran , Andre AbreuAndre Abreu , Giorgio Ivan RussoGiorgio Ivan Russo , Ege Can SerefogluEge Can Serefoglu , Andrea CocciAndrea Cocci , Kian AsanadKian Asanad , Andrey MorozovAndrey Morozov , Ioannis SokolakisIoannis Sokolakis , Afonso MorgadoAfonso Morgado , Fabio CastiglioneFabio Castiglione , Andrea SaloniaAndrea Salonia , and Giovanni E. CacciamaniGiovanni E. Cacciamani View All Author Informationhttps://doi.org/10.1097/01.JU.0001009380.44020.d3.02AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: This study assessed the ability of ChatGPT (GPT) to generate accurate, complete, clear, understandable, readable and empathetic responses to 30 scenario-based patient questions on men's health posted on an anonymous online social media forum (Reddit r/AskDocs). METHODS: Online questions about Erectile Disfunction, Premature Ejaculation, Peyronie's disease, and micropenis from March 2019 to September 2021 were collected from Reddit. 30 questions answered by verified physicians were selected and input into ChatGPT 3.5. Men's health experts rated the GPT responses for accuracy, completeness, and clarity using a Likert scale. Scores of 4 and 5 were considered positive. Public assessments of response understandability and empathy were collected via an Amazon Mechanical Turk survey. The readability of both GPT and physician responses was analyzed via Flesch-Kincaid Grade-Level (GL), SMOG Score (SMOG), Flesch Ease Score (FE), Automated Readability Index (AR), Gunning Fog Index (GF), and the Coleman-Liau Index (CL) through a validated online tool (WebFX). Statistical significance was set at p0.05) as well as similarly empathetic (Physicians: 88% vs. GPT: 87%, p>0.05). The GPT responses proved more difficult to read than physician responses on all metrics (GL: 13 (±1.0) vs 8.9 (±3.0), SMOG: 12 (±0.9) vs 8.5 (±2.1), FE 35 (±7.0) vs 61 (±15), AR 17 (±1.3) vs 12 (±3.1), GF 17 (±1.2) vs 12 (±3.1), CL 16 (±1.4) vs 11 (±2.3), p<0.05 in all readability metrics) (Figure 1). CONCLUSIONS: ChatGPT provides accurate, complete, and clear responses to men's health questions posted online, comparable in understandability and empathy to physician replies. However, ChatGPT's responses were less readable, suggesting a need for adjustments to be suitable for patient education. Download PPT Source of Funding: None © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e612 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Jacob S. Hershenhouse More articles by this author Daniel Mokhtar More articles by this author Lorenzo Storino Ramacciotti More articles by this author Severin Rodler More articles by this author Brian Hom More articles by this author Josh Ku More articles by this author John Tran More articles by this author Andre Abreu More articles by this author Giorgio Ivan Russo More articles by this author Ege Can Serefoglu More articles by this author Andrea Cocci More articles by this author Kian Asanad More articles by this author Andrey Morozov More articles by this author Ioannis Sokolakis More articles by this author Afonso Morgado More articles by this author Fabio Castiglione More articles by this author Andrea Salonia More articles by this author Giovanni E. Cacciamani More articles by this author Expand All Advertisement PDF downloadLoading ...