Natural Language Processing for risk-stratification of emergency department patients with chest pain

BACKGROUND Natural language processing (NLP) can transform unstructured clinical data into actionable insights that optimize workflows, improve patient outcomes, and reduce bias. We created a machine learning (ML) model that predicted suspicion of acute coronary syndrome (ACS) based on physician notes and assessed its capability to predict MACE when implemented within the HEART (History, ECG, Age, Risk Factors, Troponin) score. METHODS We retrospectively reviewed adults presenting with chest pain to The Ottawa Hospital emergency departments. GPT-4 produced and extracted eleven binary symptom features from physician-documented history of present illness. An extreme gradient boosting (XGBoost) classifier was trained on these features to predict ED disposition, a surrogate marker of physician suspicion of ACS (pACS). The resulting NLP-derived ACS suspicion score (nACS) replaced the History component of HEART (nHEART) and was compared with the disposition-based composite (pHEART). RESULTS Of 1011 patients (48.5% female; mean age 55.7 years), 69 (6.8%) experienced MACE within six weeks. nHEART achieved sensitivity 97.1% and specificity 53.8%; pHEART achieved 95.7% and 69.1%. pACS and troponin were the strongest multivariable predictors. GPT-4o extraction showed Fleiss' kappa > 0.8 across runs and Cohen's kappa > 0.6 versus research-assistant extraction. CONCLUSIONS Large language models can extract structured symptom features from ED notes, and an XGBoost model trained on these features can reproduce a coarse measure of physician suspicion of ACS. Clinical deployment will require further model training and testing against a prospectively documented measure of physician ACS suspicion.