Three-Dimensional Vision-Based Recognition of Guitar Chords

Abstract Artificial intelligence-powered assistants are revolutionizing the music industry by transforming how music is produced and experienced, opening new frontiers for creativity and innovation. Their application in supporting learners to play a musical instrument, however, has yet to be fully explored. Most current applications primarily focus on sound data to distinguish between different notes. This approach excludes the correct hand form, which is essential for learning to play an instrument. Furthermore, sound can be subject to background noise or compression. The current article presents a pipeline for guitar chord recognition based on 3-D data acquisition and color images. The developed method utilizes MediaPipe to estimate hand landmarks from color images and leverages depth images to retrieve real-depth information. In this study, various machine learning (ML) algorithms were compared to perform chord recognition from hand landmark information. The number of chords was limited to four, plus one class for unknown gestures. The classifier was trained with an RGB-D video data set (red, green, and blue plus depth) that features the hands of 18 subjects who performed the selected chords. The random forest (RF) classifier demonstrated remarkable performance in the classification task, achieving a balanced accuracy (BA) that exceeded 85% on unseen data under the same acquisition conditions and improving the state-of-the-art performance. Future developments will focus on expanding the number of supported chords so that the proposed approach could be used in real-world applications. Such applications might include not only guitar education but also transcription, control of sound synthesis, identification in ensembles, and other contexts where audio input alone is insufficient.