CardioCLIP: Sequential Multimodal Prediction of Echocardiographic Abnormalities and Natriuretic Peptide Elevation from ECG and Chest Radiographs

Background Diagnostic reasoning in clinical practice is inherently sequential and multimodal, with physicians integrating heterogeneous examinations under constraints such as missing data and variable test order. However, most multimodal machine learning models assume simultaneous availability of all modalities and fail to reflect this workflow. Methods We propose CardioCLIP, a sequential multimodal framework that embeds electrocardiograms (ECGs) and chest X-rays (CXRs) into a shared pathological representation space. The model consists of (1) a self-supervised embedding stage aligning ECG–CXR pairs from the same patient and day, and (2) a gated recurrent unit (GRU)-based prediction stage that estimates clinical indicators as data are sequentially observed. The model was trained on 9,458 patients from a single tertiary center and evaluated for B-type natriuretic peptide (BNP), left ventricular ejection fraction (LVEF), and E/e′. External validation was conducted using MIMIC-IV. Results CardioCLIP achieved strong cross-modal alignment and improved discrimination for selected echocardiographic abnormalities and natriuretic peptide elevation (AUC: 0.90 for BNP > 100 pg/mL, 0.87 for LVEF < 50%, and 0.80 for E/e′ > 15). It also showed discrimination for subsequent severe BNP elevation (BNP > 400 pg/mL within 3 years) with an AUC of 0.79. External validation in MIMIC-IV showed an AUC of 0.89 for NT-proBNP elevation, supporting potential transportability for this endpoint. Conclusions CardioCLIP demonstrated the feasibility of order-flexible sequential multimodal prediction. By leveraging routinely acquired ECG and CXR data, it may provide a non-invasive adjunct for identifying echocardiographic abnormalities and natriuretic peptide elevation in cardiovascular care, subject to prospective clinical validation.