Deep Multi-Modal Kernel Map Network for Music Genre Classification

Music genre classification is an important task in the music information retrieval community that aims to categorize music samples by genre; it can help to retrieve music more easily and efficiently from huge digital music resources. There is an extensive literature on music genre classification, and in this study, we solve the problem using multi-modal information, especially based on music audio and text. We propose a deep multi-modal kernel map network that learns discriminative features in a high-dimensional kernel Hilbert space by fusing the multi-modal features. For the music audio, Mel Frequency Cepstral Coefficients (MFCCs) are extracted and a pre-trained ResNet is applied to extract the features. For the texts, the pre-trained RoBERTa model is applied to extract the semantic symbolic features. In the network’s input layer, we calculate four exact/approximated elementary kernel maps from the audio and text features; in the intermediate and final layer, we progressively compute the nonlinear combination of preceding kernel maps of different modalities, followed by a fully connected layer for classification. The network can be trained end-to-end to jointly learn the combination weights between modalities and classifier parameters. We apply the proposed network on the public GTZAN dataset, multi-modal piano genre dataset, and 4MuLA dataset, and the experimental results validate the effectiveness of the proposed deep multi-modal kernel map network for music genre classification.