Haptic-Sound Analysis of Materials’ Clustering Based on Tool Tip Tapping Exploration

Tapping surfaces with a tool tip elicits both sound and haptic information. Sound and haptic information is captured by a microphone and an accelerometer attached to the tip, respectively. In relation to the task of distinguishing objects, this paper investigates the following two questions: (1) how do both the signals (sound and acceleration) help us to assess the surface individually? (2) How does the integration of both modalities affect this assessment? We approach the problem of texture assessment as an unsupervised learning problem. For this purpose, perceptual filter banks are designed based on Weber’s law of frequency perception for both modalities to extract the corresponding features. Furthermore, we introduce a symmetric KL divergence-based texture similarity metric, which helped us to compare the sound and haptic (acceleration signals) modalities. Based on our similarity metric comparisons, we argue that proximity between object types is preserved across modalities. Finally, using a permutation test-based approach, we demonstrate that the complementarity of sound domain information to the haptic domain varies depending on the type of object.