What question did this study set out to answer?

This research aims to assess the efficacy of large language models for extracting prognostic variables in prostate cancer from unstructured clinical texts.

June 14, 2026

Phenotyping Prostate Cancer in a National Health System Using Large Language Models

Key Points

This research aims to assess the efficacy of large language models for extracting prognostic variables in prostate cancer from unstructured clinical texts.
Developed prompts for 30 phenotypes using Llama 3.3 70B or GPT-4o
Data sourced from over 130 Veterans Affairs facilities between 1999 and 2025
Evaluated performance using metrics like accuracy, sensitivity, and positive predictive value.
Pathology extraction tasks demonstrated near-perfect accuracy across various reports.
For prostate biopsy, total cores and involved cores had an accuracy of 98.0% and 95.0%, respectively.
MRI pelvis scores achieved 100% accuracy for lesion dimensions, with PSMA PET/CT showing 100% PPV for nodal metastases.

Abstract

PURPOSE Large language models (LLMs) may improve extraction of prognostic variables in prostate cancer from unstructured clinical text compared with traditional, rule-based natural language processing. METHODS We used iterative prompt engineering with few-shot examples to develop LLM prompts for 30 phenotypes from prostate biopsy, radical prostatectomy (RP), and transurethral resection of the prostate (TURP) pathology reports, as well as magnetic resonance imaging (MRI) pelvis, computed tomography CT abdomen/pelvis, Tc-99m bone scan, and prostate-specific membrane antigen PSMA PET/CT reports. Data were drawn from >130 Veterans Affairs facilities (1999-2025). Inference was performed with Llama 3.3 70B or GPT-4o depending on the task. Performance was evaluated on independent test sets with metrics including overall accuracy, sensitivity, positive predictive value (PPV), negative predictive value (NPV), and macro-F1. RESULTS Pathology extraction tasks achieved near-perfect accuracy. For prostate biopsy reports, exact extraction of total cores and involved cores was highly accurate (total cores: accuracy 98.0% 95% CI, 93.0 to 99.5; involved cores: accuracy 95.0% 95% CI, 88.8 to 98.5). Performance was similarly strong for RP and TURP reports. On MRI pelvis, extraction of PIRADS scores (accuracy 98.0% 95% CI, 93.0 to 99.5), lesion locations (accuracy 100% 95% CI, 96.3 to 100), and lesion dimensions (accuracy 100% 95% CI, 96.3 to 100) was excellent. For PSMA PET/CT, PPVs were 100% (95% CI, 93.5 to 100) for nodal metastases and 97.9% (95% CI, 89.9 to 99.6) for bone metastases; Tc-99m bone scan performance was comparable. Lower PPVs were observed for nodal and bone metastases on pelvic MRI (84.2%-86.4%) and CT (88.0%-90.3%), largely due to ambiguous language in radiology report texts. CONCLUSION LLMs can reliably extract key prostate cancer phenotypes from unstructured text across multiple pathology and radiology report types. Ambiguous or indeterminate language remains the principal challenge for optimal performance.

Mark Helpful

Bookmark

Relay