Architecting Human‐AI Systems for Effective Collaboration and Oversight: Making Sense of Human/AI‐in/on/Over/Under/Along‐the‐Loop

ABSTRACT Governments around the world have written policies mandating human control over AI to ensure systems being deployed are safe and trustworthy. The core ideas behind these policies are that human control can prevent AI from making mistakes and that human collaboration with AI can leverage the unique strengths of both to create superior human‐AI systems. While the goals of these policies are laudable, they lack adequate explanations of how exactly humans are meant to have control over, or partner with, AI. To address this gap, we present a comprehensive taxonomy of human‐AI system architectures derived from synthesizing decades of human‐machine, human‐computer, and human‐robot interaction literature. Our taxonomy clarifies the conceptual inconsistencies of human‐in/on/etc.‐the‐loop terminology by analyzing prior literature through a common conceptual understanding of architecture, which we define as task allocation and the relationship between humans and AI. We identify critical gaps in existing frameworks, which have focused primarily on AI decision aids and human supervision, while neglecting emerging architectures. We also describe a four step process to use the taxonomy.