Recurrence modeling with electronic health records through natural language processing and machine learning techniques.

e18004 Background: Head and neck cancer patients with locally advanced disease requiring free flap reconstruction are at high risk for locoregional and distant recurrence yet early identification of recurrence is challenging. Unstructured clinical notes contain rich longitudinal information that may capture early signals of disease progression. We evaluated whether natural language processing (NLP)-based large language models (LLM) applied to electronic health record (EHR) notes could predict recurrence in this high-risk population. Methods: 789 patients with head and neck cancers treated with free flap reconstruction from 1997-2023 at a single institution were retrospectively analyzed. EHR pathology, radiology, and clinical notes from diagnosis through five months post-diagnosis were collected. Notes were either combined into a single aggregated document per patient or as individual notes. Any notes post-recurrence were censored. Text was vectorized using term frequency-inverse document frequency (TfidfVectorizer) and a domain-adapted contextual language model trained on clinical notes (clinicalBERT). Patients were split into training and testing cohorts prior to feature extraction. Logistic regression models were trained to predict recurrence. For per-note analyses, the note with the highest recurrence probability was used for lead-time estimation. Results: The median follow-up in the entire cohort was 39.43 months (IQR 18.53-78.23). 13.05% (103/789) patients experienced disease recurrence at a median of 8.82 months since initial diagnosis (IQR 5.83-17.67). Overall discriminative performance was modest across all models. The best performing model was the combined note TfidfVectorizer, with an AUC of 0.585. Per-note TfidfVectorizer, combined note clinicalBERT, and per-note clinicalBERT had declining AUCs of 0.580, 0.525, and 0.514 respectively. Models showed relatively high specificity but limited sensitivity for recurrent cases with a maximum F1-score of 0.28 for the combined note TfidfVectorizer. Among true-positive recurrence predictions in per-note analysis, the median lead time of predicted recurrence was 194 days (IQR 100-728) for TfidfVectorizer and 187 days (IQR 86-748) for clinicalBERT. Conclusions: NLP models applied to unstructured EHR notes demonstrated limited discrimination for recurrence prediction in head and neck cancer patients undergoing free flap reconstruction. While recurrence-associated signals were detectable months to years before diagnosis in a subset of patients, overall predictive performance was modest. Predictive performance may be improved by incorporating temporal modeling, restriction to oncology-relevant note types, aggregation of notes by date of encounter, and integration of structured clinical variables.