Modernization of a Historical Prostate Cancer Registry using Artificial Intelligence: A Tale of Two Large Language Models

PURPOSE: Since 1986, the Mayo Clinic Prostate Cancer Registry has followed over 30,000 patients who underwent radical prostatectomy. Advances in LLMs may modernize data collection, reduce human abstraction burden, and improve registry efficiency. Herein, we report the results of LLM data abstraction for surgical pathology and prostate MRI data in the Mayo Clinic Prostate Cancer Registry. MATERIAL AND METHODS: Surgical pathology and MRI reports were obtained for 3,210 and 4,739 patients, respectively, who underwent radical prostatectomy from 2010-2022. LLM prompts for 13 pathology and 20 MRI variables were developed and ran through two independent LLMs (Google Gemini Flash 1.5, 2.0, or 2.5). LLM outputs were compared to human-abstracted data. RESULTS: Median consensus LLM error rate for pathology report data was 0.09%(IQR 0.03-0.16%), compared with median human error rate of 2.81%(IQR 1.83-3.17%). Individual LLMs had threefold higher error rates compared to consensus LLM (LLM 1: 0.27%(IQR 0.09-0.34%), LLM 2: 0.30%(IQR 0.17%-0.48%)). For MRI reports, median consensus LLM error rate was 0.17%(IQR 0.00-0.68%), compared with human error rate of 1.01%(IQR 0.59-3.20%). Individual LLMs had error rates of 0.84%(IQR 0.34-2.79%) and 1.02%(IQR 0.34-1.67%). Estimated time for human data abstraction across all 136,510 data elements was 952 hours, compared with 32 hours for our LLM pipeline. CONCLUSION: LLM data abstraction employing a consensus vote across two independent LLMs consistently yielded higher accuracy than human or single-LLM abstraction. A dual-LLM consensus architecture may improve data abstraction efficiency, reduce errors, and should be considered for integration into all prospectively maintained longitudinal clinical registries.