Temporal sequence modeling for multimodal cardiovascular risk discovery

Background Cardiovascular disease prediction increasingly relies on multimodal data integration, yet identifying which data modalities causally drive risk remains an open challenge. Transformer-based temporal sequencing (TBTS) can model evolving disease trajectories across longitudinal clinical data. Purpose To develop a discovery-oriented artificial intelligence framework using transformer-based temporal sequence modeling to determine optimal feature combinations for simultaneous prediction of multiple cardiovascular outcomes at different time points. Methods We build a multi-task temporal sequencing Generative Pre-trained Transformer (GPT)-based architecture on 80,140 UK Biobank participants using cardiac magnetic resonance (CMR) and electrocardiographic phenotypes, polygenic risk score and clinical data collected at multiple time points. Our approach models disease progression as temporal sequences (GPTts) across multiple timepoints. The architecture employs shared representation learning with different feature combinations to simultaneously predict the incidence of a primary composite endpoint of major adverse cardiovascular events (MACE), and secondary endpoints - heart failure, stroke, myocardial infarction and atrial fibrillation - over a 10-year horizon. The model was trained, validated and tested in the 70%, 15% and 15%, respectively. Results In the test set, the GPT framework achieved optimal overall discrimination considering all five cardiovascular outcomes simultaneously (area under the ROC curve [AUC] 0.923–0.963; area under the precision-recall curve [AUPRC] 0.672–0.865; Brier score 0.028). The temporal sequence modeling approach captured evolving disease trajectories, achieving AUC 0.963 for myocardial infarction, 0.960 for stroke, 0.935 for heart failure, 0.948 for atrial fibrillation, and 0.923 for MACE using clinical features alone. Adding CMR imaging further improved cardiac-specific outcomes (heart failure: 0.946, +1.1%; atrial fibrillation: 0.959, +1.1%), demonstrating the model's ability to integrate multimodal temporal information. Attention mechanism analysis of the optimal configuration (clinical + CMR) demonstrated that the transformer learned to integrate temporal disease patterns, allocating balanced attention across clinical features (51.4%) and CMR imaging (38.3%), capturing complementary prognostic information from evolving phenotypes. Conclusions Transformer-based temporal sequence modeling enables simultaneous prediction of multiple cardiovascular outcomes with excellent discrimination. By capturing evolving disease trajectories across longitudinal clinical assessments, our multi-task architecture achieves state-of-the-art performance for parallel risk stratification of the primary and secondary endpoints. This approach demonstrates how deep learning can identify which data modalities causally drive cardiovascular risk through temporal sequence analysis.