Obey, refrain, or contravene: disobedience as a basis for responsible autonomy

In human-agent teams, there are situations when agents ought to disobey human commands due to various factors, including situational misunderstandings by the command-issuer or rapidly changing conditions. As a reliable member of a team, an agent should be able to evaluate a situation, avoid unnecessary risks, and potentially prevent errors in human judgment stemming from cognitive biases, miscommunication, or incomplete information. Prior work on this topic focuses on disobedience through refraining from execution (i.e., inaction). Here we additionally consider disobedience by contravention—the execution of an alternative course of action, and show that how an agent can disobey affects when it is appropriate for that agent to disobey. To this end, we create a conceptual model of intelligent disobedience. We present results from simulations of our model and discuss the implications of intelligent disobedience in collaborative settings.