Abstract We consider the reality of deploying artificial intelligence (AI) in safety-critical systems, such as autonomous vehicles, medical diagnoses and weather forecasting. Our discussion is grounded in the mathematical nature of AI systems, including how an AI's mathematical properties relate to its benefit and risk profile. Benefits include the ability to learn models from data even when no physical model exists, increased automation and enhanced speed compared with traditional approaches. Risks of AI include its opaque (mis-)understanding of the world, failures on out-of-distribution inputs, its insatiable appetite for data and computation and the ongoing challenge of aligning the AI's objectives with human values. Such risks are potentially manageable with clear-eyed expectations, and our hope in this work is to clarify what can be expected. This article is part of the theme issue ‘Safe, secure and robust AI for safety-critical systems’.
Tamara G. Kolda (Thu,) studied this question.