DARPA's explainableAI(XAI) program: A retrospective

Summary of Defense Advanced Research Projects Agency's (DARPA) explainable artificial intelligence (XAI) program from the program managers' and evaluator's perspective. Defense Advanced Research Projects Agency (DARPA) formulated the explainable artificial intelligence (XAI) program in 2015 with the goal to enable end users to better understand, trust, and effectively manage artificially intelligent systems. In 2017, the 4-year XAI research program began. Now, as XAI comes to an end in 2021, it is time to reflect on what succeeded, what failed, and what was learned. This article summarizes the goals, organization, and research progress of the XAI program. Dramatic success in machine learning has created an explosion of new AI capabilities. Continued advances promise to produce autonomous systems that perceive, learn, decide, and act on their own. These systems offer tremendous benefits, but their effectiveness will be limited by the machine's inability to explain its decisions and actions to human users. This issue is especially important for the United States Department of Defense (DoD), which faces challenges that require the development of more intelligent, autonomous, and reliable systems. XAI will be essential for users to understand, appropriately trust, and effectively manage this emerging generation of artificially intelligent partners. The problem of explainability is, to some extent, the result of AI's success. In the early days of AI, the predominant reasoning methods were logical and symbolic. These early systems reasoned by performing some form of logical inference on (somewhat) human readable symbols. Early systems could generate a trace of their inference steps, which could then become the basis for explanation. As a result, there was significant work on how to make these systems explainable.1-5 Yet, these early AI systems were ineffective; they proved too expensive to build and too brittle against the complexities of the real world. Success in AI came as researchers developed new machine learning techniques that could construct models of the world using their own internal representations (eg, support vectors, random forests, probabilistic models, and neural networks). These new models were much more effective, but necessarily more opaque and less explainable. The year 2015 was an inflection point in the need for XAI. Data analytics and machine learning had just experienced a decade of rapid progress.6 The deep learning revolution had just begun, following the breakthrough ImageNet demonstration in 2012.6, 7 The popular press was alive with animated speculation about Superintelligence8 and the coming AI Apocalypse.9, 10 Everyone wanted to know how to understand, trust, and manage these mysterious, seemingly inscrutable, AI systems. 2015 also saw the emergence of initial ideas for providing explainability. Some researchers were exploring deep learning techniques, such as the use of deconvolutional networks to visualize the layers of convolutional networks.11 Other researchers were pursuing techniques to learn more interpretable models, such as Bayesian Rule Lists.12 Others were developing model-agnostic techniques that could experiment with a machine learning model—as a black box—to infer an approximate, explainable model, such as LIME.13 Yet, others were evaluating the psychological and human-computer interaction aspects of the explanation interface.13, 14 DARPA spent a year surveying researchers, analyzing possible research strategies, and formulating the goals and structure of the program. In August 2016, DARPA released DARPA-BAA-16-53 to call for proposals. The stated goal of explainable artificial intelligence (XAI) was to create a suite of new or modified machine learning techniques that produce explainable models that, when combined with effective explanation techniques, enable end users to understand, appropriately trust, and effectively manage the emerging generation of AI systems. The target of XAI was an end user who depends on decisions or recommendations produced by an AI system, or actions taken by it, and therefore needs to understand the system's rationale. For example, an intelligence analyst who receives recommendations from a big data analytics system needs to understand why it recommended certain activity for further investigation. Similarly, an operator who tasks an autonomous system needs to understand the system's decision-making model to appropriately use it in future missions. The XAI concept was to provide users with explanations that enable them to understand the system's overall strengths and weaknesses; convey an understanding of how it will behave in future/different situations; and perhaps permit users to correct the system's mistakes. The XAI program assumed an inherent tension between machine learning performance (eg, predictive accuracy) and explainability, a concern that was consistent with the research results at the time. Often the highest performing methods (eg, deep learning) were the least explainable and the most explainable (eg, decision trees) were the least accurate. The program hoped to create a portfolio of new machine learning and explanation techniques to provide future practitioners with a wider range of design options covering the performance-explainability trade space. If an application required higher performance, the XAI portfolio would include more explainable, high-performing, deep learning techniques. If an application required more explainability, XAI would include higher performing, interpretable models. The program was organized into three major technical areas (TAs), as illustrated in Figure 1: (a) the development of new XAI machine learning and explanation techniques for generating effective explanations; (b) understanding the psychology of explanation by summarizing, extending and applying psychological theories of explanation; and (c) evaluation of the new XAI techniques in two challenge problem areas: data analytics and autonomy. The original program schedule consisted of two phases: phase 1, Technology Demonstrations (18 months); and phase 2, Comparative Evaluations (30 months). During phase 1, developers were asked to demonstrate their technology against their own test problems. During phase 2, the original plan was to have developers test their technology against one of two common problems (Figure 2) defined by the government evaluator. At the end of phase 2, the developers were expected to contribute prototype software to an open source XAI toolkit. In May 2017, XAI development began. Eleven research teams were selected to develop the Explainable Learners (TA1) and one team was selected to develop the Psychological Models of Explanation. Evaluation was provided by the Naval Research Lab. The following summarizes those developments and the final state of this work at the end of the program. An interim summary of the XAI developments at the end of 2018 is given in Gunning and Aha.15 The program anticipated that researchers would examine the training process, model representations, and, importantly, explanation interfaces. Three general approaches were envisioned for model representations. Interpretable model approaches would seek to develop ML models that were inherently more explainable and more introspectable for machine learning experts. Deep explanation approaches would leverage deep learning or hybrid deep learning approaches to produce explanations in addition to predictions. Finally, model induction techniques would create approximate explainable models from more opaque, black-box models. Explanation interfaces were expected to be a critical element of XAI, connecting a user to the model to enable them to understand and interact with the decision making process. As the research progressed, 11 XAI teams explored a number of machine learning approaches, such as tractable probabilistic models16 and causal models and explanation techniques such as state machines generated by reinforcement learning algorithms,17 Bayesian teaching,18 visual saliency maps,19-24 and network and GAN dissection.24-26 Perhaps the most challenging and most unique contributions came from the combination of machine learning and explanation techniques27 to conduct well-designed psychological experiments to evaluate explanation effectiveness.28-31 As the program progressed, we also gained a more refined understanding of the spectrum of users and development timeline (Figure 3). The program structure anticipated the need for a grounded psychological understanding of explanation. One team was selected to summarize current psychological theories of explanation to assist the XAI developers and the evaluation team. This work began with an extensive literature survey on the psychology of explanation and previous work on explainability in AI.32 Originally, this team was asked to (a) produce a summary of current theories of explanation, (b) develop a computational model of explanation from those theories, and (c) validate the computational model against the evaluation results from the XAI developers. Developing computational models proved to be a bridge too far, but the team did gain a deep understanding of the area and successfully produced descriptive models. These descriptive models were critical to supporting the effective evaluation approaches, which involved carefully designed user studies, carried out in accordance with DoD human subject research guidelines. Figure 4 illustrates a top-level descriptive model of the XAI explanation process. Evaluation was originally envisioned to be based on a common set of problems, within the data analytics and autonomy domains. However, it quickly became clear that it would be more valuable to explore a variety of approaches across a breadth of problem domains. In order to evaluate the performance in the final year of the program, the evaluation team, led by Eric Vorm, PhD, of the US Naval Research Laboratory (NRL), developed an explanation scoring system (ESS). This scoring system provided a quantitative mechanism for assessing the designs of XAI user studies in terms of technical and methodological appropriateness and robustness. The ESS enabled the assessments of multiple elements of each user study, including the task, domain, explanations, explanation interface, users, hypothesis, data collection, and analysis to ensure that each study met the high standards of human subject research. XAI evaluation measures are shown in Figure 5, and include functional measures, learning performance measures, and explanation effectiveness measures. The DARPA XAI program demonstrated definitively the importance of carefully designing user studies in order to accurately evaluate the effectiveness of explanations in ways that directly enhance appropriate use and trust by human users, and appropriately support human-machine teaming. Often times, multiple types of measures (ie, performance, functionality, and explanation effectiveness) will be necessary to evaluate the performance of an XAI algorithm. XAI user study design can be tricky and the DARPA XAI program discovered that the most effective research teams were ones that featured diverse teams with cross-disciplinary expertise (ie, computer science combined with human-computer interaction and/or experimental psychology, etc.). The XAI program explored many approaches, as shown in Table 1. Interactive debugger interface for visualizing poisoned training datasets. Work is applied on the IARPA TrojAI dataset.33 Establishing objective/quantitative criteria to assess value of explanations for ML models34 CNN-based one-shot detector, using network dissection to identify the most salient features41 Explanations produced by heat maps and text explanations42 Human-machine common ground modeling Indoor navigation with a robot (in collaboration with GA Tech) Video Q&A Human-assisted one-shot classification system by identifying the most salient features Three major evaluations were conducted during the program: one during phase 1 and two during phase 2. In order to evaluate the effectiveness of XAI techniques, researchers on the program designed and executed user studies. User studies are still the gold standard for assessing explanations. There were approximately 12 700 participants in user studies carried out by XAI researchers, including approximately 1900 supervised participants, where the individual was guided through the experiment by the research team (eg, in person or on Zoom) and 10 800 unsupervised participants, where the individual self-guided through the experiment and was not actively guided by the research team (eg, Amazon Mechanical Turk). In accordance with policy for all US DoD funded human subjects research, each research protocol was reviewed by a local Institutional Review Board (IRB) and then a DoD human research protection office reviewed the protocol and the local IRB findings. As mentioned earlier, there seemed to be a natural tension between learning performance and explainability. However, throughout the course of the program, we found evidence that explainability can improve performance. From an intuitive perspective, training a system to produce explanations provides additional supervision, via additional loss functions, training data, or other mechanisms, that encourages a system to learn more effective representations of the world. While this may not be true in all cases and significant work remains to characterize when explainable techniques will be more performant, it provides hope that future XAI systems can be more performant than current systems while meeting user needs for explanations. There currently is no universal solution to XAI. As discussed earlier, different user types require different types of explanations. This is no different from what we face interacting with other humans. Consider, for example, a doctor needing to explain a diagnosis to a fellow doctor, a patient, or a medical review board. Perhaps future XAI systems will be able to automatically calibrate and communicate explanations to a specific user within a large range of user types, but that is still significantly beyond the current state of the art. One of the challenges in developing XAI is measuring the effectiveness of an explanation. DARPA's XAI effort has helped develop foundational technology in this area, but much more needs to be done, including drawing more from the human factors and psychology communities. Measures of explanation effectiveness need to be well established, well understood, and easily implemented by the developer community in order for effective explanations to become a core capability of ML systems. UC Berkeley's result21 demonstrating that advisability, the ability for an AI system to take advice from a user, improves user trust beyond explanations is intriguing. Certainly, users will likely prefer systems where they can quickly correct the behavior of a system in the same ways that humans can provide feedback to each other. Such advisable AI systems that can both produce and consume explanations will be key to enabling closer collaborations between humans and AI systems. Close collaboration is required across multiple disciplines including computer science, machine learning, artificial intelligence, human factors, and psychology, among others, in order to effectively develop XAI techniques. This can be particularly challenging, as researchers tend to focus on a single domain and often need to be pushed to work across domains. Perhaps in the future a XAI-specific research discipline will be created at the intersection of multiple current disciplines. Toward this end, we have worked to create an XAI Toolkit, which collects the various program artifacts (eg, code, papers, reports, etc.) and lessons learned from the 4-year DARPA XAI program into a central, publicly accessible location (https://xaitk.org/).48 We believe the toolkit will be of broad interest to anyone who deploys AI capabilities in operational settings and needs to validate, characterize, and trust AI performance across a wide range of real-world conditions and application areas. Today, we have a more nuanced, less dramatic, and, perhaps, more accurate understanding of AI than we had in 2015. We certainly have a more accurate understanding of the possibilities and the limitations of deep learning. The AI apocalypse has faded from an imminent danger to a distant curiosity. Similarly, the XAI program has produced a more nuanced, less dramatic, and, perhaps, more accurate understanding of XAI. The program certainly acted as a catalyst to stimulate XAI research (both inside and outside of the program). The results have produced a more nuanced understanding of XAI uses and users, the psychology of XAI, the challenges of measuring explanation effectiveness, as well as producing a new portfolio of XAI ML and HCI techniques. There is certainly more work to be done, especially as new AI techniques are developed that will continue to need explanation. XAI will continue as an active research area for some time. The authors believe that the XAI program has made a significant contribution by providing the foundation to launch that endeavor. David Gunning (now retired) is a three-time DARPA program manager, who created and managed the XAI program from its inception in 2016 to its mid-point in 2019. His portfolio of DARPA research programs made significant contributions to the development of AI over the past 25 years. He led the Personalized Assistant that Learns (PAL) program, which produced the technologies behind Apple's Siri. His Command Post of the Future (CPoF) program was later adopted by the US Army as their Command and Control system for use during the Iraq and Afghanistan conflicts. Between DARPA tours, David served in senior positions at Facebook AI, Palo Alto Research Center, Vulcan Inc, Cycorp and co-founded SET Corp. Eric Vorm, PhD, is a cognitive systems engineer and serves as the Deputy Director for the Laboratory for Autonomous Systems Research at the US Naval Research Laboratory in Washington, DC. Dr Vorm led the evaluation team for the DARPA Explainable AI program, and led the development of the first comprehensive criteria for the evaluation of explanations generated by machine learning. Dr Vorm's research focuses on the design of intelligent systems to achieve ideal human-machine teaming, with special emphasis on the role of transparency and explainability in supporting appropriate trust, safety, and reliability in high-risk, time-sensitive operational domains. Jennifer Yunyan Wang, PhD, is a computational neuroscientist with a special focus on AI. As Systems, Engineering and technical Assistance (SETA) contractor to DARPA, she provided technical support and expertise to several programs including XAI, L2M, GARD, and AIE RED. After finishing postdoctoral fellowships in experimental neuroscience at Johns Hopkins University and the Food and Drug Administration, Jennifer joined Quantitative Scientific Solutions in 2018 as a consultant for government R&D and think tanks including IARPA and Center for Security and Emerging Technology. Matt Turek, PhD, joined DARPA's Information Innovation Office (I2O) as a program manager in July 2018 and took over as program manager of the XAI program in 2019. His portfolio also includes the Media Forensics (MediFor), Semantic Forensics (SemaFor), and Machine Common Sense (MCS) programs, as well as the Reverse Engineering of Deceptions (RED) AI Exploration. His research interests include computer vision, machine learning, artificial intelligence, and their application to problems with significant societal impact. Prior to his position at DARPA, Turek led a team at Kitware Inc developing computer vision technologies including large scale behavior recognition and modeling, object detection and tracking, activity recognition, normalcy modeling, and anomaly detection. Data sharing is not applicable to this article as no new data were created or analyzed in this editorial.