Designing and Evaluating a Clinical Decision Support System to Assess Mild Cognitive Impairment

Explainable artificial intelligence (XAI) is increasingly used in clinical decision support systems (CDSS) to improve the interpretability and trustworthiness of machine learning (ML) predictions in healthcare. While prior work has examined how healthcare professionals (HCPs) interpret various explanation methods, limited research has investigated how these explanations function in systems that incorporate novel data sources like game-based digital biomarkers. This paper extends our previous work by presenting an improved, interactive web-based CDSS designed to support HCPs in assessing the risk of Mild Cognitive Impairment (MCI). The system integrates ML predictions from gameplay data with data-centric, feature-importance, and counterfactual explanations to assist diagnosis and case-finding. We conducted a mixed-methods study with 51 HCPs to evaluate the system's understandability (objective and perceived), perceived usefulness, perceived ease of use, and trust. Participants gave consistently positive ratings across the self-reported perceived constructs, which were found to be strongly correlated. HCPs generally perceived the explanations as useful for clinical reasoning and reported that they understood and trusted the system, while acknowledging an initial learning curve. However, the evaluation of objective understanding was more mixed and correlated only weakly with the perceived constructs. In particular, participants encountered challenges in understanding game-based digital biomarkers, interpreting feature importances, combining local and global explanations, and understanding details of the underlying ML models. These findings highlight the importance of explanation designs that foster both positive user perceptions and accurate comprehension.