Mp24-01 Integrating Natural Language Processing With Chatgpt to Improve Quality of Artificial Intelligence Summaries of Prostate Cancer Consultations

You have accessJournal of UrologyHealth Services Research: Practice Patterns, Quality of Life and Shared Decision Making II (MP24)1 May 2024MP24-01 INTEGRATING NATURAL LANGUAGE PROCESSING WITH CHATGPT TO IMPROVE QUALITY OF ARTIFICIAL INTELLIGENCE SUMMARIES OF PROSTATE CANCER CONSULTATIONS Sanjay K. Das, Nadine Friedrich, Michael Luu, Stephen J. Freedland, Brennan Spiegel, and Timothy J. Daskivich Sanjay K. DasSanjay K. Das , Nadine FriedrichNadine Friedrich , Michael LuuMichael Luu , Stephen J. FreedlandStephen J. Freedland , Brennan SpiegelBrennan Spiegel , and Timothy J. DaskivichTimothy J. Daskivich View All Author Informationhttps://doi.org/10.1097/01.JU.0001008860.46052.c4.01AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Large Language Model-based artificial intelligence such as ChatGPT has potential to improve physician-patient communication by providing summaries of key data from treatment consultations to patients and providers. However, ChatGPT summaries from raw consultation transcripts are often unfocused and lack detail. Pre-processing with natural language processing (NLP) models to identify topic-specific content may improve quality of ChatGPT summaries. We investigated whether NLP models identifying topic-specific content used in combination with ChatGPT improve the topic concordance and quality of risk information of ChatGPT summaries about these topics from prostate cancer (PC) consultations. METHODS: We recorded and transcribed 42 consultations for early-stage PC. Validated NLP models extracted information on tradeoffs for decision making (cancer prognosis (CP), life expectancy (LE), erectile dysfunction (ED), irritative urinary symptoms (IUS), urinary incontinence (UI)). We prompted ChatGPT3.0 to summarize each topic using NLP-identified text at NLP-based probability thresholds for topic concordance (50%-90%). ChatGPT summaries were reviewed for proportion of sentences related to the topic, binary topic concordance (total vs not), and whether risk information was quantified. Poisson and logistic regression were used to assess the association of NLP probability threshold with these outcomes. RESULTS: We analyzed 1,050 ChatGPT summaries. The proportion of topic concordant sentences significantly increased as NLP probability threshold increased for all five topics. For each 10% increase in NLP probability threshold, proportion of sentences related to ED increased by 27% (Incidence Rate Ratio(IRR) 1.27, 95%CI 1.2-1.3), 36% for UI (IRR 1.36, 95%CI 1.3-1.4), 41% for IUS (IRR 1.41; 95%CI 1.3-1.5), 56% for LE (IRR 1.56, 95%CI 1.4-1.7), and 34% for CP (IRR 1.34, 95%CI 1.3-1.4). Odds of binary topic concordance was similarly associated with NLP probability threshold for all five topics. Odds of the ChatGPT summary containing a quantified risk estimate increased only for LE (OR 1.57, 95%CI 1.1–2.3). CONCLUSIONS: Use of NLP pre-processing improves topic concordance of ChatGPT summaries of key tradeoffs for PC decision making and quality of risk data for select topics. Download PPT Source of Funding: Dr. Das was supported by the Patient-Centered Outcomes Research Training in Urologic and Gynecologic Cancers (PCORT UroGynCan): T32CA251072. Dr. Friedrich was supported by the NIH grant T32HL116273 © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e391 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Sanjay K. Das More articles by this author Nadine Friedrich More articles by this author Michael Luu More articles by this author Stephen J. Freedland More articles by this author Brennan Spiegel More articles by this author Timothy J. Daskivich More articles by this author Expand All Advertisement PDF downloadLoading ...