User-Centered Development of Explanation User Interfaces for AI-Based CDSS: Lessons Learned from Early Phases

This paper reports lessons learned during the early phases of the user-centered design process for an explanation user interface for an AI-based clinical decision support system for the intensive care unit. This paper focuses on identifying and verifying physicians’ explanation needs in a multi-center, multi-country project. The explanation needs identified through context analysis and user requirements prioritization in an initial center differed from those identified through questionnaire responses from N= 9 physicians after a multi-center project workshop. These results highlight the caution that should be taken when eliciting explanation needs during the user-centered design process.