65 Integrating pathology reports into multimodal machine learning models to predict thyroid cancer recurrence

Objectives/Goals: The objective of this study was to evaluate the performance of multimodal machine learning (ML) models trained to predict differentiated thyroid cancer (DTC) recurrence using clinical data combined with novel natural language processing (NLP) derived features extracted from patient cytopathology and surgical pathology reports. Methods/Study Population: This was a retrospective study of adult thyroid cancer patients treated at an academic medical center. Patients were classified as having cancer recurrence or no recurrence. NLP features were extracted from cytopathology and surgical pathology reports using Term Frequency–Inverse Document Frequency (TF-IDF), latent Dirichlet allocation (LDA), and a zero-shot large language model (LLM) classification. 5 multimodal ML models were trained to predict cancer recurrence utilizing a combination of NLP and LLM features and clinical variables. Model performance was evaluated using area under the receiver operating characteristic curve (ROC-AUC) and precision recall area under the curve (PR-AUC). The top performing model was optimized with a 5-fold cross-validation. Feature importance was calculated. Results/Anticipated Results: 480 patients with differentiated thyroid cancer diagnosed on surgical pathology were included in this study. The baseline model (clinical variables only) had a F1-score of 0.52 and an AUC of 0.53. The optimized gradient boosting model utilizing all features (EMR, LDA, TF-IDF, and LLM) had a F1-score of 0.87 and an AUC of 0.86. Topic words and themes from the patient cytopathology and surgical pathology reports were generated using LDA. Topic themes in cytopathology reports include malignancy, lymph node evaluation, and molecular testing. Topic themes in surgical pathology reports include histologic subtype, orientation of nodule, and intraoperative biopsy. The LDA themes of malignancy and histologic subtype ranked the highest in terms of feature importance. Discussion/Significance of Impact: Multimodal models utilizing novel NLP features derived from unstructured pathology reports may enable improved prediction of recurrence in patients with DTC. Our optimized model demonstrated that 4 of the top 6 highest features were LDA topics. Topic modeling may be a valuable tool to extract relevant information from unstructured clinical notes.