A large language model agent for multimodal evaluation of cardiovascular disease

Background Artificial intelligence (AI) is rapidly expanding our analytical capabilities for diagnosis and management of cardiovascular disease. Unlike clinicians who synthesize information from multiple data sources in real-world settings, clinical AI algorithms have typically relied on single data modalities and limit their applications to narrowly defined tasks. Purpose We developed an AI agent powered by a large language model (LLM) and deep learning tools to retrieve information from multimodal data sources including clinical notes, electrocardiography (ECG), and echocardiography for classification of cardiovascular conditions. Methods The agent is built upon Gemini 1.5 as the core LLM and uses retrieval augmented generation to interpret clinical notes, with a deep learning classifier to diagnose atrial fibrillation (Afib) from ECGs, and a vision language model to process echocardiography images (Figure 1). The agent is deployed with an interactive user interface and validated on unimodal and multimodal tasks using clinical notes, ECGs, and echocardiography studies from MIMIC-IV. Ground truth labels for all experiments were manually abstracted by a cardiologist and disease classification experiments were all conducted in single-turn and single-word answers (yes/no) for objective assessment of performance. Results The agent demonstrated a success rate of >99% in invoking the appropriate tools in response to modality-specific queries (N= 2,850 queries). It achieved an overall accuracy of 93-98% in classification of atrial fibrillation, heart failure with reduced ejection fraction, coronary artery disease, and diabetes from clinical notes (sensitivity 86-94% and specificity 95-100%, N = 100 for each condition). The agent showed excellent discrimination of atrial fibrillation from ECGs (AUROC = 0.94, AUPRC = 0.63, N= 2000) and a strong correlation with the expert cardiologist in estimating the left ventricular ejection fraction from echocardiography images (Pearson correlation coefficient = 0.88, N = 220). Using multimodal inputs, the agent achieved an enhanced diagnostic accuracy of 92-98% when clinical notes and ECGs were concordant and 85-91% when the two modalities were discordant for or against a diagnosis of atrial fibrillation (Figure 2). Conclusion Our agent combines the conversational features of language models with task-specific capabilities of deep learning tools, offering an interactive and highly accurate approach for evaluation of cardiovascular conditions. The multimodal capabilities of an agentic framework provide the opportunity to improve diagnostic accuracy by synthesizing information across cardiovascular data domains.