Haptic Music Feedback through Audio Decomposition

This study presents a novel approach to improving musical haptic wearables by applying deep learning models for music source separation and pitch estimation. The system consists of a haptic vest and a pair of haptic gloves, designed to spatially convey musical elements throughout the body. By isolating instruments from an audio file and recognizing their pitches, our system provides intuitive haptic feedback that discretely represents each instrument's performance. Specifically, we map the piano to the gloves, the bass to the back of the vest, and the drums to the front of the vest. Then, we conducted a comparative study with a conventional audio-to-haptic method. The results showed that users experienced improved clarity, intuitiveness, and comprehension, indicating enhanced understanding of musical structure.