Fine-Tune the Pretrained ATST Model for Sound Event Detection

Sound event detection (SED) often suffers from the data deficiency problem. Recent SED systems leverage the large pretrained self-supervised learning (SelfSL) models to mitigate such restriction, where the pretrained models help to produce more discriminative features for SED. However, the pretrained models are regarded as a frozen feature extractor in most systems, and fine-tuning of the pretrained models has been rarely studied. In this work, we study the fine-tuning method of the pretrained models for SED. We introduce frame-level audio teacher-student transformer model (ATST-Frame), our newly proposed SelfSL model, to the SED system. ATST-Frame was especially designed for learning frame-level representations of audio signals and obtained state-of-the-art (SOTA) performances on a series of downstream tasks. We then propose a fine-tuning method for ATST-Frame using both (in-domain) unlabelled and labelled SED data. Our experiments show that, the proposed method overcomes the overfitting problem when fine-tuning the large pre-trained network, and our SED system obtains new SOTA results of 0.587/0.812 PSDS1/PSDS2 on the DCASE challenge task 4 dataset.