Transformer-based multimodal representation learning for blind acoustic classification

Context recognition and rare event classification in unlabeled audio represent critical challenges in both academic and industrial domains. These technologies hold significant potential for applications such as IoT-based predictive maintenance, surveillance systems (e.g., crime scene analysis or aviation incident investigations), and urban management (e.g., traffic monitoring). While recent research predominantly focuses on supervised methods using labeled datasets, unsupervised approaches remain constrained by limitations in accuracy and stability. This paper introduces a novel AI framework that synergizes a self-supervised audio representation model with a knowledge-enhanced language model to sequentially classify environmental sound types and rare events in unlabeled audio. Our methodology leverages an enhanced Masked Modeling Duo (M2D) —pretrained on datasets including FSD50K, ESC-50, and UrbanSound8K—to process mixed audio inputs, generate discriminative sound labels, and classify unlabeled data. The MPNet-base-v2 then analyzes these labels, establishing semantic relationships to infer environmental context and rare event categories. When evaluated on the TUT-SED dataset (DCASE 2017 Task 2), the framework achieves F-scores of 96% (baby cry) , 98% (glass break) , and 91% (gunshot) for acoustic event detection and classification. For blind environmental classification (DCASE 2017 Task 1), the accuracy of the top five predictions is 95%, and that of the top one prediction is 87%. Although memory constraints may impact classification quality when processing long-duration audio recordings, robustness tests demonstrate resilience to noise. Event classification maintains stable performance at a 0.1 SNR (-10 dB) , with average F-scores of 95% for baby cry , 97% for glass break , and 85% for gunshot . The scene classification accuracy also decreases modestly, reaching 89% at an SNR of -10 dB, but remains comparable to that under noise-free conditions. An extended methodology application to additional datasets—specifically, DESED 2019 for multiple acoustic event classification (MAEC) and TAU Urban Acoustic Scenes 2020 Mobile for acoustic scene classification (ASC)—further substantiated the algorithmic generalization capability across diverse acoustic contexts. Such a capability can be used for more downstream environmental sound studies, e.g., sound separations. The implementation is publicly accessible on GitHub: https://github.com/sharontan6217/TMRL.