Mp24-03 Development and Validation of a Natural Language Processing System to Identify Information on Key Tradeoffs in Prostate Cancer Treatment Consultations

You have accessJournal of UrologyHealth Services Research: Practice Patterns, Quality of Life and Shared Decision Making II (MP24)1 May 2024MP24-03 DEVELOPMENT AND VALIDATION OF A NATURAL LANGUAGE PROCESSING SYSTEM TO IDENTIFY INFORMATION ON KEY TRADEOFFS IN PROSTATE CANCER TREATMENT CONSULTATIONS Timothy Daskivich, Michael Luu, Rebecca Gale, Dmitry Khodyakov, Stephen Freedland, and Brennan Spiegel Timothy DaskivichTimothy Daskivich , Michael LuuMichael Luu , Rebecca GaleRebecca Gale , Dmitry KhodyakovDmitry Khodyakov , Stephen FreedlandStephen Freedland , and Brennan SpiegelBrennan Spiegel View All Author Informationhttps://doi.org/10.1097/01.JU.0001008860.46052.c4.03AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Men with prostate cancer face treatment decisions requiring careful consideration of risks and rewards of therapy. However, patient retention of information and inconsistent physician risk communication are barriers to informed shared decision making (SDM). We sought to develop and validate natural language processing (NLP) models for identifying and assessing quality of information communicated on key tradeoffs during consultations. METHODS: We analyzed 42 initial consultation transcripts of men with prostate cancer across a multidisciplinary cohort of providers. Sentences were coded for whether they contained information on key tradeoffs: cancer prognosis (CP), life expectancy (LE), erectile dysfunction (ED), urinary incontinence (UI), and irritative urinary tract symptoms (LUTS). Diagnostic metrics and receiver-operator curve (ROC) analysis assessed performance of NLP models at identifying tradeoff-specific content compared with manual coding in the training dataset. The best performing model was validated in a separate dataset. The association between NLP probability of topic relevance and informational quality was assessed using linear regression in the validation dataset. RESULTS: The dataset included 17,195 sentences, with 75% reserved for training and 25% for validation. The Random Forest model had the highest area under the curve (AUC) in ROC analysis in the training dataset. Using this model, AUC values in the validation dataset for identifying sentences related to ED, UI, LUTS, CP, and LE were 0.96 (95%CI 0.94–0.98), 0.96 (95%CI 0.92–0.99), 0.99 (95%CI 0.99–0.99), 0.91 (95%CI 0.88–0.94), and 0.86 (95%CI 0.74–0.95), respectively (Figure 1). Sensitivity and specificity ranged from 0.71–0.95 and 0.86–0.96 respectively. NLP probability of topic relevance was associated with higher informational quality for all topics (p60%, >70%, and >75% respectively. CONCLUSIONS: NLP models can accurately identify high-quality information on key tradeoffs from treatment consultations. This information can be reported back to providers and patients after the consultation to enhance quality of risk communication and SDM. Download PPT Source of Funding: This work was supported by Career Development Award (K08 CA230155 to TJD) from the National Cancer Institute © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e392 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Timothy Daskivich More articles by this author Michael Luu More articles by this author Rebecca Gale More articles by this author Dmitry Khodyakov More articles by this author Stephen Freedland More articles by this author Brennan Spiegel More articles by this author Expand All Advertisement PDF downloadLoading ...