Understanding User Trust and Expectations in Answerable Sociotechnical Systems

Today, we see an increase in the number of systems where Artificial intelligence and Autonomous systems (AI/AS) are included in decision-making processes even in high-stake domains such as healthcare. It is often not clear how to take responsibility appropriately and provide restitution when significant harms happen, especially when organizations use opaque AI/AS to make decisions. We focus on the notion of responsibility as answerability, which demands that individuals have a duty to supply their “reasons and justifications” for an action, or respond to inquiries about why they acted in a particular manner. We investigate the trust towards organizations that use mediator agents, people’s expectations from such agents, perceived benefits and risks of using this new intervention to enable answerability in organizations. Our findings indicate that providing explanations and offering remedies for harm through mediator agents does not affect the level of trust individuals have in organizations.