Mp07-14 Development of a Generative Artificial Intelligence Data Pipeline to Automate the Capture of Unstructured Mri Data for Prostate Cancer Care

You have accessJournal of UrologySurgical Technology & Simulation: Artificial Intelligence I (MP07)1 May 2024MP07-14 DEVELOPMENT OF A GENERATIVE ARTIFICIAL INTELLIGENCE DATA PIPELINE TO AUTOMATE THE CAPTURE OF UNSTRUCTURED MRI DATA FOR PROSTATE CANCER CARE Anobel Y. Odisho, Andrew W. Liu, William A. Pace, Robert Krumm, Janet E. Cowan, Peter R. Carroll, and Matthew R. Cooperberg Anobel Y. OdishoAnobel Y. Odisho , Andrew W. LiuAndrew W. Liu , William A. PaceWilliam A. Pace , Robert KrummRobert Krumm , Janet E. CowanJanet E. Cowan , Peter R. CarrollPeter R. Carroll , and Matthew R. CooperbergMatthew R. Cooperberg View All Author Informationhttps://doi.org/10.1097/01.JU.0001008728.41882.d7.14AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Radiology reports are stored as unstructured text in the electronic health record, rendering the data inaccessible. Large language models (LLM) and generative artificial intelligence (genAI) are powerful tools for analyzing and generating text. While many web-based LLM tools are available, they require manual entry of each prompt and are not approved for use with protected health information (PHI). Our objective was to develop an automated LLM pipeline to extract unstructured data from prostate MRI reports and compare the accuracy to manually abstracted data. METHODS: The University of California, San Francisco has deployed a HIPAA-compliant internal LLM tool using OpenAI's GPT-4 platform approved for PHI use. We developed a detailed prompt instructing the LLM to extract specified data elements when presented with prostate MRI reports, and to output the results in a structured, computer readable format (JavaScript Object Notation, JSON). Using this prompt, a data pipeline was built using the OpenAI Application Programming Interface (API) to automatically extracted prostate volume, membranous urethra length, number of concerning lesions, locations of lesions, PI-RADS score of lesions, extracapsular extension, seminal vesicle invasion, and the presence of enlarged lymph nodes. We compared accuracy and calculated Cohen's Kappa between the LLM-based model pipeline and manual abstraction. RESULTS: 228 prostate MRI reports with existing manually abstracted data were analyzed. Median accuracy of GenAI across all tested elements was 89.5% (IQR 89.0-91.5) compared to gold standard. Median concordance of GenAI to gold standard was 0.68 (IQR 0.56-0.91). LLM extracted accuracy was 96% for prostate volume, 89% for PSA density, and 92% for length of membranous urethra. The LLM performed best when extracting specific continuous values (i.e. prostate volume or membranous urethral length), and worse with binary or categorical values (i.e. presence of seminal vesicle involvement). CONCLUSIONS: LLMs can be used to flexibly extract text data from imaging reports with high accuracy and low up-front programming requirements. This can be an effective and scalable mechanism for medical information extraction to power clinical, research, and operational use cases. Source of Funding: N/A © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e110 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Anobel Y. Odisho More articles by this author Andrew W. Liu More articles by this author William A. Pace More articles by this author Robert Krumm More articles by this author Janet E. Cowan More articles by this author Peter R. Carroll More articles by this author Matthew R. Cooperberg More articles by this author Expand All Advertisement PDF downloadLoading ...