Explainable AI in Healthcare: Systematic Review of Clinical Decision Support Systems

ABSTRACT This systematic review examines the evolution and current landscape of eXplainable Artificial Intelligence (XAI) in Clinical Decision Support Systems (CDSS), highlighting significant advancements and identifying persistent challenges. Utilising the PRISMA protocol, we searched major indexed databases such as Scopus, Web of Science, PubMed, and the Cochrane Library, to analyse publications from January 2000 to April 2024. This timeframe captures the progressive integration of XAI in CDSS, offering a historical and technological overview. The review covers the datasets, application areas, machine learning models, explainable AI methods, and evaluation strategies for multiple XAI methods. Analysing 68 articles, we uncover valuable insights into the strengths and limitations of current XAI approaches, revealing significant research gaps and providing actionable recommendations. We emphasise the need for more public datasets, advanced data treatment methods, comprehensive evaluations of XAI methods, and interdisciplinary collaboration. Our findings stress the importance of balancing model performance with explainability and enhancing the usability of XAI tools for medical practitioners. This research provides a valuable resource for healthcare professionals, researchers, and policymakers seeking to develop and evaluate effective, ethical decision-support systems in clinical settings.