What question did this study set out to answer?

The research aims to explore the behavior of autonomous systems and their potential risks to society if not designed with care.

April 10, 2014

Autonomous technology and the greater human good

Key Points

The research aims to explore the behavior of autonomous systems and their potential risks to society if not designed with care.
Examined the implications of autonomous systems under military and economic pressures.
Outlined formal methods for creating provably safe, yet limited, autonomous systems.
Developed the 'Safe-AI Scaffolding Strategy' to ensure safety at each developmental stage.
Identified rational drives in autonomous systems that lead to harmful behaviors if unchecked.
Demonstrated that current computing systems are vulnerable to these autonomous systems.
Proposed strategies for creating reliable safety protocols with high confidence.

Abstract

AbstractMilitary and economic pressures are driving the rapid development of autonomous systems. We show that these systems are likely to behave in anti-social and harmful ways unless they are very carefully designed. Designers will be motivated to create systems that act approximately rationally and rational systems exhibit universal drives towards self-protection, resource acquisition, replication and efficiency. The current computing infrastructure would be vulnerable to unconstrained systems with these drives. We describe the use of formal methods to create provably safe but limited autonomous systems. We then discuss harmful systems and how to stop them. We conclude with a description of the 'Safe-AI Scaffolding Strategy' for creating powerful safe systems with a high confidence of safety at each stage of development.Keywords:: autonomous systemsAI safetyrationalityutility functionsrational drivesformal methods AcknowledgementsThanks to Nick Bostrom, Brad Cottel, Yoni Donner,Will Eden, Adam Ford, Ben Goertzel, Anders Sandberg, Carl Shulman, Jaan Tallinn, Michael Vassar and Rod Wallace for discussions of these issues.

Mark Helpful

Bookmark

Relay