A Systematic Review of Explainable Convolutional Neural Networks for Image Classification

ABSTRACT Explainable AI (XAI) is crucial for fostering human trust in deep neural network (DNN) predictions, particularly in tasks like image classification. Multiple surveys exist on XAI methodologies, however, the practical usability and reproducibility of these methods remain largely unexplored. This paper addresses this gap by conducting a systematic survey of recent XAI papers published in leading computer vision and AI conferences and journals. We categorize these works, identify prevalent datasets and evaluation metrics, and analyse the associated code repositories. Our analysis reveals that almost 95% of the surveyed codebases are research prototypes rather than published releases, and a concerning majority of two‐thirds of them exhibit inconsistencies with their corresponding publications. These findings highlight the challenges in benchmarking new XAI methods against existing ones and explain the slow adoption of state‐of‐the‐art research in real‐world applications. This paper aims to underscore the importance of releasing well‐documented, readily usable code alongside XAI research to foster a more robust and reproducible ecosystem, ultimately facilitating the development and deployment of trustworthy AI systems. The results of this study are presented on an interactive website Interactive‐XAI.