Designing Human‐AI Architectural Robustness to Inherently Brittle AI

Machine learning based systems have made impressive advances in recent years, but they are still brittle, often failing when encountering new and unexpected changes to the operating environment. Humans are often partnered with AI‐enabled systems to negate this brittleness, but human operators may struggle with changes to the scale of the operating environment, as cognitive fatigue can decrease the effectiveness of human oversight. Through a simulation of a notional minefield traversal system, we show how humans and AI are integrated together affects system outcomes. We propose that evaluations of human‐AI systems should consider different human‐AI integration architectures and that AI‐enabled systems should also deploy continuous monitoring to detect shifts in the operating environment which may cause AI system performance to faulter. Finally, system design should enable human operators to have sufficient situational awareness, which may require scaling up the number of human operators available.