Transformer-based multimodal data integration for enhanced prediction of major adverse cardiovascular events in patients with non-obstructive coronary artery disease

Background Risk stratification in non-obstructive coronary artery disease (CAD) remains challenging due to heterogeneity across clinical and imaging data. Multimodal machine learning approaches offer the potential to improve prognostic accuracy. Objective To develop and validate a transformer-based multimodal model that integrates electronic health records (EHR), blood tests, electrocardiography (ECG), and computed tomography coronary angiography (CTCA) features to predict major adverse cardiovascular events (MACE) in patients with non-obstructive CAD. Methods This retrospective cohort study was conducted at the participating hospital from January 2014 to October 2024. The study included consecutive patients undergoing CTCA, with those requiring revascularization excluded. Across 7940 eligible CTCA assessments, 729 were excluded for prior events, and 452 for revascularization, resulting in 6,759 eligible for analysis. Integrated data modalities included diagnostic codes from EHRs, laboratory tests, digitized ECG signals processed via the DigitHeart pipeline, and CTCA-derived metrics such as plaque volume, tissue composition, stenosis severity, fractional flow reserve, and fat attenuation index. These data were integrated using a transformer architecture featuring multi-level embedding and temporal encoding to fuse the diverse multimodal information. The primary endpoint was a composite of major adverse cardiovascular events (MACE), defined as cardiovascular death, acute coronary syndrome, heart failure hospitalization, or ventricular arrhythmia. Model performance was evaluated using the area under the receiver operating characteristic curve (AUC), and risk stratification was assessed with Cox proportional hazards models. Results Single-modal models yielded AUCs of 0.640 for diagnostic codes, 0.690 for blood tests, 0.610 for ECG, and 0.680 for CTCA. Combining CTCA with EHR or blood tests modestly improved performance, yielding AUCs of 0.710 and 0.700, respectively. The full multimodal transformer model achieved a superior AUC of 0.760. Patients classified as high risk by the model demonstrated a significantly increased risk of MACE, with a hazard ratio of 1.40 (95% CI 1.00-1.96; p = 0.049). Conclusions A transformer-based multimodal integration approach significantly improves MACE prediction in non-obstructive CAD compared with single-modality assessments, supporting more precise, personalized risk stratification. Prospective validation in external cohorts is warranted before clinical implementation.