An MR-HRI Framework for Mobile Devices to Communicate Force Intent and Receive Visual Force Feedback

As robots and humans start to share common spaces and perform collaborative tasks, it has become critical to facilitate information exchange between them for communicating and interpreting each other’s intentions. By overlaying virtual objects on a view of the physical world, mixed reality (MR) technology offers a compelling approach for designing innovative models of human–robot interaction (HRI). For robot manipulators, mobile MR frameworks that allow a user to communicate a goal position for the robot’s end effector have been widely studied. However, HRI applications that may require other relevant information for the manipulator to complete more complex tasks remain unexplored. Thus, we propose an MR-enhanced HRI framework, deployed on a touchscreen tablet, that utilizes a virtual arrow object to communicate force intent (i.e., location, direction, and magnitude) to the manipulator and provide visual force feedback to the user. To evaluate the system performance and user experience, we conducted a user study with 25 participants who used a manipulator robot to complete four insertion subtasks, reporting a task success score of 96%, a usability overall mean score of 4.35 out of 5, and a low task load index of 21.49 out of 100. The results show that the MR-HRI framework is intuitive to operate, allowing users to successfully perform assigned tasks by effectively communicating their intentions through the virtual arrow.