August 1, 2025

When Explanations Differ: A Qualitative Study of Clinical Views on Explainable AI (XAI) Methods in Healthcare (Preprint)

Key Points

Clinicians value explainable AI for decision-making in healthcare, especially when facing complex cases.
Discrepancies in AI explanations are less concerning if predictions are accurate and insights actionable.
The research involved semi-structured interviews with 14 clinicians in Australian critical care settings.
Implementing trustworthy AI tools requires aligning explanations with clinicians' reasoning to enhance trust.

Abstract

BACKGROUND AI-driven clinical decision support (CDS) tools offer promising solutions for healthcare delivery by optimising resource allocation, detecting deterioration and enabling early interventions. However, adoption remains limited due to insufficient validation and a lack of transparency. eXplainable AI (XAI) provides users with insights into AI-driven recommendations, but discrepancies in explanations, known as the "Disagreement Problem", can undermine trust and at worst, lead to poor clinical decisions. OBJECTIVE This study explores the perspectives of clinicians from Australian critical care settings on XAI and the impact of discrepancies in AI-generated explanations on decision-making. METHODS Qualitative data were collected using semi-structured interviews with 14 clinical experts, incorporating scenario-based exercises, and analysed using inductive thematic analysis. RESULTS Key factors influencing trust in XAI are identified, and the role of explainability in AI-driven tools are highlighted. Explainability was considered valuable, especially in unfamiliar situations or complex decisions, if the explanations were clear, plausible, and actionable. Discrepancies in explanations generated by different XAI methods are not the primary concern for clinicians, provided the AI’s prediction was accurate and the explanations offered actionable insights aligning with their mental model. CONCLUSIONS This study has identified design recommendations and implementation strategies for developing trustworthy, user-centric XAI-supported CDS tools. It also highlights that discrepancy between different explanations is not inherently problematic, provided the explanations are consistent with clinicians’ reasoning. Recommendations made highlight the importance of aligning the design and implementation of AI tools with clinicians’ needs to enhance trust, mitigate risks, and promote successful adoption for improved patient outcomes.

اسأل الذكاء الاصطناعي

Bookmark