Key points are not available for this paper at this time.
Artificial intelligence (AI) has the potential to transform the delivery of health care by automating complex tasks that traditionally required substantial training and expertise. Contrary to concerns that fully automated AI will replace health-care providers, 1Obermeyer Z Emanuel EJ Predicting the future — big data, machine learning, and clinical medicine. N Engl J Med. 2016; 375: 1216-1219Crossref PubMed Scopus (1375) Google Scholar a more likely scenario is that providers will increasingly interact with task-specific and domain-specific AI systems across a continuum of automation. The level of AI autonomy is broadly accepted to be crucial to the risk-assessment of AI algorithms, yet its definition and classification is poorly defined. 2American Medical AssociationAugmented intelligence in health-care: payment and regulation. https: //www. ama-assn. org/system/files/2019-08/ai-2019-board-report. pdfDate: 2019Date accessed: July 31, 2020Google Scholar, 3Abràmoff MD Tobey D Char DS Lessons learned about autonomous AI: finding a safe, efficacious, and ethical path through the development process. Am J Ophthalmol. 2020; 214: 134-142Summary Full Text Full Text PDF Scopus (39) Google Scholar, 4US Food and Drug AdministrationPublic workshop—evolving role of artificial intelligence in radiological imaging. https: //www. fda. gov/medical-devices/workshops-conferences-medical-devices/publicworkshop-evolving-role-artificial-intelligence-radiologicalimaging-02252020-02262020Date: 2020Date accessed: July 31, 2020Google Scholar In this Comment, we present a nuanced discussion of autonomy in medical AI to improve the understanding of the effect, risks, and clinical value of such systems. An AI's autonomous function is critical to defining its intended use, which is how the developer intends the algorithm to be applied and guides its development, validation, regulation, and deployment. Medical AI algorithms are typically classified as either assistive (non-autonomous) or fully autonomous. 4US Food and Drug AdministrationPublic workshop—evolving role of artificial intelligence in radiological imaging. https: //www. fda. gov/medical-devices/workshops-conferences-medical-devices/publicworkshop-evolving-role-artificial-intelligence-radiologicalimaging-02252020-02262020Date: 2020Date accessed: July 31, 2020Google Scholar This distinction broadly captures the risk imparted by autonomous AI function, and is particularly instructive in establishing liability in cases of medical error, and determining reimbursement for medical services incorporating AI systems. However, this binary classification does not capture the spectrum of AI autonomy, which has important clinical implications and should be intentionally considered when approaching these technologies. The relationship between autonomous systems and risk has been discussed extensively in the automobile and aviation industries. 5US Department of TransportationPreparing for the future of transportation. Automated vehicles 3. 0. https: //www. transportation. gov/sites/dot. gov/files/docs/policy-initiatives/automated-vehicles/320711/preparing-future-transportation-automated-vehicle-30. pdfDate: 2018Date accessed: July 31, 2020Google Scholar, 6US Federal Aviation AdministrationOperational use of flight path management systems: final report of the Performance-based operations Aviation Rulemaking Committee/Commercial Aviation Safety Team, Flight Deck Automation Working Group. https: //www. faa. gov/aircraft/aircert/designₐpprovals/humanfactors/media/OUFPMSReport. pdfDate: 2013Date accessed: July 31, 2020Google Scholar The US Department of Transportation identifies five levels of automation, ranging from no automation to full automation. 7SAE InternationalJ3016B: taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. https: //www. sae. org/standards/content/j3016₂01806/Date: 2018Date accessed: July 31, 2020Google Scholar A similar model for medical AI systems would improve risk stratification and development of evaluation standards in the health-care setting. Therefore, within the binary umbrellas of assistive and autonomous AI algorithms, we suggest a model of graded autonomy in medical AI systems based on central determinants of autonomy-related risk: the responsible agent for medical event monitoring and response, the expected and available backup decision-maker, and the specificity of the medical domain, health-care system, and population that the algorithm is intended to operate within (figure). The performance of automated AI systems in the real world and previously untested clinical scenarios is largely unknown, demanding thorough clinical evaluation. Preclinical testing, including external validation using datasets reflective of populations the model is intended for, should be done on all models to confirm generalisability to their intended population. 4US Food and Drug AdministrationPublic workshop—evolving role of artificial intelligence in radiological imaging. https: //www. fda. gov/medical-devices/workshops-conferences-medical-devices/publicworkshop-evolving-role-artificial-intelligence-radiologicalimaging-02252020-02262020Date: 2020Date accessed: July 31, 2020Google Scholar, 8Kelly CJ Karthikesalingam A Suleyman M Corrado G King D Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019; 17: 195Crossref PubMed Scopus (486) Google Scholar However, such validation is insufficient to fully understand clinical value even for level 1 autonomous AI systems because, although preclinical testing is often carried out under ideal conditions, AI systems will need to function together with other software, hardware, and highly variable end-users in the real-world setting. Therefore, any AI system should undergo, at minimum, closely supervised pilot testing in the clinic setting to understand its real-world function in a controlled fashion. These early pilot tests should be overseen by clinicians who could serve as backup during initial deployment, as well as data scientists who understand the complexities and limitations of deployed algorithms. This strategy provides the opportunity to incorporate end-user training and feedback, serving the critical role of optimising the user interface and establishing trust in the system. Clinical AI algorithms currently undergo US Food and Drug Administration approval via medical device pathways, although AI-specific regulatory frameworks are in development. 2American Medical AssociationAugmented intelligence in health-care: payment and regulation. https: //www. ama-assn. org/system/files/2019-08/ai-2019-board-report. pdfDate: 2019Date accessed: July 31, 2020Google Scholar, 9US Food and Drug AdministrationProposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML) -based software as a medical device (SaMD). https: //www. fda. gov/media/122535/downloadDate accessed: July 31, 2020Google Scholar, 10US Food and Drug AdministrationDeveloping a software precertification program: a working model. https: //www. fda. gov/downloads/MedicalDevices/DigitalHealth/DigitalHealthPreCertProgram/UCM629276. pdf? utmcampaign=Digital%20Health%20Update%3A%20New%20FDA%20Pre-Cert%20Working%20Model%20Now%20Available 214: 134-142Summary Full Text Full Text PDF Scopus (39) Google Scholar Given the black box nature of many AI algorithms and concerns regarding generalisability, such technologies should first undergo safety testing as assistive technologies before being evaluated as autonomous algorithms. Rigorous outcome-based testing with performance metrics that address safety, efficacy, and equity during clinical evaluation is necessary for ethical system development. Assessing the effect of autonomous AI systems on different populations, for example by incorporating a diagnosability endpoint for diagnostic systems, will help ensure that these technologies improve, rather than potentiate, disparities. 3Abràmoff MD Tobey D Char DS Lessons learned about autonomous AI: finding a safe, efficacious, and ethical path through the development process. Am J Ophthalmol. 2020; 214: 134-142Summary Full Text Full Text PDF Scopus (39) Google Scholar Particularly for autonomous systems, ongoing quality control and post-market surveillance to monitor continued system performance will provide additional confidence. 4US Food and Drug AdministrationPublic workshop—evolving role of artificial intelligence in radiological imaging. https: //www. fda. gov/medical-devices/workshops-conferences-medical-devices/publicworkshop-evolving-role-artificial-intelligence-radiologicalimaging-02252020-02262020Date: 2020Date accessed: July 31, 2020Google Scholar Because clinical evaluation will need to be repeated for each new intended medical domain and population, level 5 automation in which a system can function fully autonomously in any medical setting is unlikely to be safely achieved in the near term. Automated AI algorithms might affect health-care provider performance in unexpected ways, for example by improving their independent performance by providing training or by reducing the value of their independent skills by the absence of practice and diminished attention. The potential for an AI algorithm to affect health-care delivery, through loss of access, must be considered for each application, and explicit examination of this effect should be incorporated into clinical evaluation. Further, simulation training of system failures might need to be adopted to sustain provider skills if key clinical areas become routinely automated. Finally, particularly for systems in which clinician engagement might be reduced but not removed completely, incorporating built-in alerts for provider engagement and awareness can support human performance. Transparency is essential to stakeholder trust in automated medical AI systems, and will need to be prioritised for these technologies to be adopted. Robust clinical evidence is an essential aspect of developing trust in these technologies but will likely be insufficient for widespread adoption because of the unfamiliarity of these systems and concerns regarding ongoing performance, workforce replacement, and cybersecurity. Therefore, as systems become more autonomous, education on their function and maintenance, as well as safety specifications that are readable to a lay person, should be made available. AI autonomy is critical to determining liability. In the case of assistive algorithms, the clinician ultimately makes the clinical judgement or decision, and so should be held liable. The American Medical Association recommends that AI developers should be held liable for medical errors made by autonomous AI algorithms. 2American Medical AssociationAugmented intelligence in health-care: payment and regulation. https: //www. ama-assn. org/system/files/2019-08/ai-2019-board-report. pdfDate: 2019Date accessed: July 31, 2020Google Scholar However, for level 3 models, liability should depend on whether an error was caused directly by the AI, or by the clinician after they were involved as the backup operator. A risk-based approach to autonomy in medical AI systems should inform regulation, clinical validation, and clinician engagement. As key stakeholders, it is the duty of providers to understand the ways in which AI systems might function autonomously, and what our expected role and responsibility will be in interacting with them. DSB reports consultancy fees from Agios Pharmaceuticals; HJWLA reports consultancy fees and stock from Onc. Ai; and RHM is a scientific advisory board member for AstraZeneca and ViewRay, and reports a research grant from ViewRay, all outside of the submitted work.
Bitterman et al. (Mon,) studied this question.