567 Background: Selecting adjuvant therapy after nephrectomy for renal cell carcinoma (RCC) requires reviewing scattered electronic health records (EHR) and performing accurate eligibility and risk assessments. Large Language Models (LLMs) may assist physicians by extracting key clinical data and generating evidence-based recommendations between active surveillance and adjuvant pembrolizumab. Methods: We selected RCC patients who underwent nephrectomy at Mayo Clinic in Arizona, Florida, and Minnesota between December 2021 and September 2025. Data extraction from EHRs was performed by a HIPAA-compliant LLM (GPT-5 and Gemini 2.0-Flash-001) and compared to human labels. Data on patients’ demographics, medical history, pathology, laboratory results, and imaging were collected. GPT-5 generated treatment recommendations using a structured prompt that incorporated patients’ features and relevant clinical information that is available to clinicians. Concordance between the human labels and LLM was evaluated for extraction. For treatment recommendations, agreement between the LLM’s output and physician’s documented recommendation in clinical notes was assessed, with disagreements reviewed independently by two blinded genitourinary oncologists. Results: The dataset included 85 patients, median (range) age of 65 (34-84) years; 24.7% were females. Patients who declined adjuvant treatment for personal reasons were excluded. LLM achieved an extraction accuracy of 95% for presenting symptoms, 97% for BMI and ECOG, and > 99% for pathology variables. Concordance between LLM and physician treatment recommendations was 84.7% (72/85). Among 13 discrepant cases, one oncologist agreed with the LLM in 9 cases and the other in 8. The main reasons for disagreement between the physicians and the LLM were that the LLM favored active surveillance for patients with declining renal function, whereas physicians still considered adjuvant pembrolizumab appropriate in such cases. Additionally, the LLM demonstrated hesitancy to recommend pembrolizumab for patients with low performance status. Overall, the 13 discrepant cases were nuanced and challenging, with disagreement observed between the two oncologists themselves in three instances. Conclusions: LLMs demonstrated high accuracy in extracting clinically relevant variables and generating objective, evidence-based adjuvant treatment recommendations for RCC after nephrectomy. Importantly, LLM did not generate any inappropriate or potentially harmful recommendations. Our pipeline can serve as a supportive decision tool to enhance consistency in post-nephrectomy management and assist physicians in choosing between active surveillance and adjuvant pembrolizumab. However, final treatment decisions should remain the responsibility of the treating physician after comprehensive patient evaluation.
Nahhat et al. (Sun,) studied this question.