Validating Large Language Models for Identifying Pathologic Complete Responses After Neoadjuvant Chemotherapy for Breast Cancer Using a Population-Based Pathologic Report Data

Abstract In the context of breast cancer management, the accurate determination of pathologic complete response (pCR) from large narrative pathology reports is pivotal for cancer outcome and survivorship studies. Leveraging the Large Language Models (LLMs) in digital pathology, our study developed and validated methods for identifying pCR from pathology reports of 351 breast cancer patients who underwent neoadjuvant chemotherapy. The optimum method demonstrated a sensitivity of 100.0% (95%CI: 100.0-100.0%), positive predictive value of 84.0% (95%CI: 70.0-96.8%), and F1 score of 91.3% (95%CI: 81.5–98.1%). These algorithms, integrating diverse LLMs, exhibited superior performance compared to traditional machine learning models. Our findings suggest LLMs hold significant potential utility in clinical pathology for extracting critical information from textual data.