Beyond Visual and Force Feedback: Role of Vibrotactile and Auditory Cues in Robot Teleoperated Assembly

Reliable detection of contact states, such as the “mating” of connectors, is crucial for high-quality teleoperated assembly. Conventional systems relying solely on visual and continuous force feedback often fail to convey these discrete high-frequency transients due to the limited high-frequency rendering capabilities. This study investigates the effectiveness of augmenting visual and force feedback with vibrotactile and auditory cues for detecting connector mating. We conducted three experiments: (1) a mating detection task using recorded multimodal data (N=10), (2) a modality contribution analysis (N=10), and (3) a real-time robot connector insertion task (N=10). Results from the real-time task demonstrated that the proposed multimodal feedback significantly reduced the maximum contact force exerted after mating compared to the baseline visual-force condition (p<0.001), thereby enhancing physical safety. Furthermore, vibrotactile and auditory cues were found to be redundant yet complementary, providing robust cues even when one modality is compromised. Although subjective mental workload increased due to sensory integration, the significant improvement in detection clarity and safety justifies the multimodal approach. We conclude that providing transient vibrotactile and auditory cues is a highly effective strategy for compensating for the limitations of conventional force feedback in teleoperated assembly.