Digital Twin-empowered Model-Mediated Teleoperation using Multimodality Data with Signed Distance Fields

In human-in-the-loop teleoperation systems, the Quality of Task Perception (QoTP) plays an important role, significantly influencing the overall Quality of Experience (QoE). QoTP characterizes the depth of understanding a user possesses regarding remote environments and tasks. To enhance this comprehension, this paper introduces an innovative geometric modeling approach employing Truncated Signed Distance Fields (TSDF) tailored for Model-Mediated Teleoperation (MMT) systems. Our approach combines haptic and visual cues to provide real-time updates to environmental changes. To ensure optimal responsiveness, we devise a model updating scheme that achieves an effective update frequency. Capitalizing on the capabilities of a Digital Twin, the MMT system presented not only uses the existing knowledge of the remote environment but also records its dynamic alterations. Through comprehensive experimentation evaluating both the geometric modeling and the teleoperation framework, our findings underscore the robustness and applicability of the introduced geometric modeling technique.