What is Wrong With Automation Bias?

Abstract This article examines the ethical and moral implications of automation bias in high-stakes decision-making contexts. Drawing on empirical studies, we distinguish between weak automation bias, where users follow system’s automated cues (or its silence) without consulting readily accessible evidence that contradicts them, and strong automation bias, where users follow such cues (or their absence) even when they are aware of such evidence. While weak automation bias, in our view, resembles automation-based complacency and is plausibly associated with negligence on the part of the human operator, strong automation bias reveals an excessive and unwarranted transfer of trust from operators to automated systems which results in epistemic deference of the former to the prompts of the latter. We argue that what is ethically and morally troubling about this form of deference, is that it interferes with the exercise of the operators’ autonomous agency as well as with their duty to exercise human judgment in high-stakes decision-making contexts. To mitigate these effects, we discuss two design-based tools introducing epistemic friction - Reflection Machines (RMs) and defeaters - which ultimately aim at cultivating critical trust in the interaction between human operators and decision-support systems.