Explainable AI for HCI: a systematic review of a decade of studies on XAI tools

Abstract Despite their increasing adoption, understanding black-box models that underpin AI technologies is not trivial. To address this challenge, explainable AI (XAI) tools have received increased interest, albeit their HCI exploration is limited. We report a systematic review of 142 papers focused on the design, use, or evaluation of XAI tools, and in particular, the types of XAI tools, their exploration, the type of data input and output, the type of explanation, available XAI interfaces, and evaluation of XAI tools. Our outcomes highlight a large number of XAI tools, albeit a small number is commonly used, most often by AI experts as users rather than novice users. Our findings led to four research, design, and methodological implications, including the call for more HCI-driven user interaction with XAI tools, for designing richer portfolios of user-data interaction scenarios for XAI tools, and of novel XAI user interfaces, and for leveraging hybrid methods and measures for XAI tools evaluation.