MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Classification Tasks

This paper presents a benchmark study of MIDI-domain music classification using the mask language modeling approach of the Bidirectional Encoder Representations from Transformers (BERT). Specifically, with five public-domain datasets of single-track polyphonic piano MIDI files, we pre-train a 12-layer Transformer model using the BERT approach and fine-tune it for four downstream classification tasks. These include two note-level classification tasks---melody extraction and velocity prediction, and two sequence-level classification tasks---artist classification and emotion classification. Compared to strong recurrent neural network (RNN)-based baselines, the BERT approach does lead to higher classification accuracy, with less than 10 epochs of fine-tuning. Ablation studies further show that the pre-training remains effective even if the MIDI data of the downstream tasks are not seen during the pre-training stage, and that freezing the self-attention layers at the fine-tuning stage only slightly degrades the performance. For reproducibility, we share the weights of our pre-trained and fine-tuned models.