MusiQAl: Music Question-Answering through Audio-Video Fusion

This paper introduces MusiQAl, a comprehensive Music Question-Answering dataset comprising 310 music performance videos paired with 11,830 question-answer sets, each set annotated by expert musicologists. Combining information from audio and video enhances the description of a music performance. Audio captures sound nuances, while video shows stage presence and expressive music gestures. These gestures convey crucial musical information like dynamics, phrasing, and emotion. The audio-visual music question-answering task exemplifies leveraging both modalities to address music-related queries, illustrating the effectiveness of this multimodal approach. However, many existing datasets may not fully capture the diversity of musical genres or the nuanced aspects of musical performance. Including audio-visual captions, the dataset features various musical instruments from at least 18 countries across all five continents and encompasses multiple genres, including instrumental, dance, and vocal performances. Our dataset showcases the diversity of music performances and establishes a link between question-answer pairs and studies on music cognition. We employ advanced techniques in information fusion for multimodal music processing, showcasing the efficacy of MusiQAl on state-of-the-art music question-answering models. Our research underscores the critical role of human-annotated data and highlights the necessity for a new taxonomy of music question templates.