Failure-Aware Adaptation for Free-Hand Interaction in Mixed Reality

Free-hand interaction in Mixed Reality (MR) can become difficult as the interaction technique, target geometry, and spatial context vary. We present FAIR-MR, a failure-aware adaptive interaction system that responds to unsuccessful interaction through geometric interventions. Two user studies characterize interaction difficulty across gestures and spatial conditions, and then compare Static, Precomputed, and Online Bandit strategies for selecting interventions. Interaction success varied across gestures, with target geometry affecting interaction techniques differently. The adaptation strategies produced distinct intervention and interaction patterns. Within Online Bandit, intervention estimates became increasingly differentiated with experience, and greater pre-decision differentiation was associated with subsequent recovery. However, this policy development did not produce higher aggregate success than the other strategies. These findings reveal how interaction conditions, intervention strategies, and learned policy behavior shape adaptive interaction. FAIR-MR contributes a system-level approach connecting interaction difficulty, intervention selection, interface changes, transition costs, and recovery.