Unveiling the black box: imperative for explainable AI in cardiovascular disease prevention

Dalakoti et al. have a positive attitude towards the application of artificial intelligence (AI) in cardiovascular disease (CVD) prevention in Singapore, especially emphasizing the potential of AI tools such as CardioSight and CHAMP in identifying high-risk individuals and implementing preventive measures.1Dalakoti M. Djohan A. Ong J. et al.Incorporating AI into cardiovascular diseases prevention–insights from Singapore.Lancet Reg Health West Pac. 2024; 48101102PubMed Google Scholar However, a core issue is missing from its argument: the explainability and transparency of these AI systems. Given the increasing penetration of AI technology into medical decision-making, it has become a top priority to uncover the mystery of the "black box" and ensure that the recommendations provided by these systems can be understood and trusted by both physicians and patients alike. The lack of explainability in AI systems poses significant challenges in the medical domain, because medical decisions directly impact patient health and safety. Physicians need to comprehend the logical basis of AI recommendations to communicate effectively with patients and make prudent decisions.2Amann J. Blasimme A. Vayena E. et al.Explainability for artificial intelligence in healthcare: a multidisciplinary perspective.BMC Med Inform Decis Mak. 2020; 20: 310Crossref PubMed Scopus (562) Google Scholar For example, when CardioSight identifies that a patient is at high risk for CVD, the physician must be able to understand and explain the factors contributing to this assessment, otherwise, they may miss the early intervention opportunities because of the uncertainty of AI recommendations. In parallel, patients have the right to know how their health data is being used and how AI algorithms arrive at specific conclusions. The opaque systems will erode patients' trust, hinder patient-physician communication, and impede shared decision-making.3Ploug T. Holm S. The four dimensions of contestable AI diagnostics-A patient-centric approach to explainable AI.Artif Intell Med. 2020; 107101901Crossref PubMed Scopus (86) Google Scholar Must be aware that the success of AI-driven CVD prevention depends not only on the accuracy of predictions but also on the transparency of the decision-making process. Therefore, researchers and developers should give priority to the development of explainable AI models in healthcare, which involves designing algorithms that can provide clear, interpretable explanations for their predictions and recommendations.4Tjoa E. Guan C. A survey on explainable artificial intelligence (XAI): toward medical XAI.IEEE Trans Neural Netw Learn Syst. 2021; 32: 4793-4813Crossref PubMed Scopus (792) Google Scholar Techniques such as feature importance analysis, rule extraction and counterfactual explanation to unravel the complex process of AI decision-making.5Henry K.E. Kornfield R. Sridharan A. et al.Human–machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.NPJ Digit Med. 2022; 5: 97Crossref PubMed Scopus (71) Google Scholar Increasing investment in explainable AI aims to foster trust, facilitate effective communication, and ensure that AI tools such as CardioSight and CHAMP are powerful, transparent and accountable for realizing the fundamental change of cardiovascular disease prevention and patient care model. Authors declare no competing interests. Incorporating AI into cardiovascular diseases prevention–insights from SingaporeImproved upstream primary prevention of cardiovascular disease (CVD) would enable more individuals to lead lives free of CVD. However, there remain limitations in the current provision of CVD primary prevention, where artificial intelligence (AI) may help to fill the gaps. Using the data informatics capabilities at the National University Health System (NUHS), Singapore, empowered by the Endeavour AI system, and combined large language model (LLM) tools, our team has created a real-time dashboard able to capture and showcase information on cardiovascular risk factors at both individual and geographical level- CardioSight. Full-Text PDF Open AccessUnveiling the black box: imperative for explainable AI in cardiovascular disease (CVD) prevention–author replyIn their commentary entitled "Unveiling the Black Box: Imperative for Explainable AI in Cardiovascular Disease (CVD) Prevention", Wu et al. rightly emphasise the need for explainability and transparency of artificial intelligence (AI) systems.1 We agree resonantly. These are essential for healthcare workforce buy-in, accountability to patients and delivery of equitable care. In our earlier article in this journal, we discussed the ways in which AI is being incorporated into a CVD prevention service in the National University Heart Centre Singapore (NUHCS).