Explainable AI-based techniques for disease classification: a systematic review and future work

Early and accurate diagnosis of critical diseases is essential for timely intervention and improved patient outcomes. While state-of-the-art machine learning (ML) and deep learning (DL) models have demonstrated high performance in disease classification, their adoption in real-world clinical workflows remains limited. A major barrier is the lack of transparency, trust, and systematic evaluation of explainability in black-box AI systems. This study aims to provide a comprehensive understanding of explainability in imaging-based XAI, where substantial evidence exists, while also including radiomics, multimodal, and selected non-imaging studies in black-box AI architectures for disease classification. Specifically, we systematically analyze methodological choices in explainable AI (XAI), evaluate their impact on interpretability, identify key challenges, and outline future research directions. In addition, we assess the clinical readiness of XAI-based decision support systems using the Clinician-informed XAI evaluation checklist with metrics (CLIX-M), aligned with the XAI 2.0 manifesto. A systematic literature review was conducted following Kitchenham and Charters’ guidelines. We reviewed peer-reviewed studies published between January 2019 and December 2025 that applied XAI-enabled ML or DL models for disease classification. Data were extracted and synthesized across datasets, preprocessing strategies, model architectures, XAI techniques, validation practices, and deployment-relevant criteria. Out of 5,005 ML/DL-based disease classification studies, 105 employed XAI-based architectures. Across these, we identified 49 datasets or data sources (including imaging, radiomics, and tabular), 131 ML/DL models spanning 12 architectural families, and 27 distinct XAI techniques. Most studies were retrospective (80%), only 8.5% conducted formal XAI evaluations, 21.0% performed external validation, and merely 10% shared source code. Although a wide range of XAI-based ML and DL architectures have been proposed for disease classification, particularly in imaging-based applications, as well as selected non-imaging tabular and multimodal applications, their clinical readiness remains limited. Key shortcomings include the lack of clinician-in-the-loop or human-centered validation, the absence of standardized XAI evaluation protocols, poor reproducibility and code availability, insufficient robustness testing, weak integration into clinical workflows, and the complexity of generated explanations. These gaps highlight the need for rigorously evaluated, clinically grounded, and reproducible XAI frameworks to enable trustworthy deployment in real-world healthcare settings.