Enhancing Spatial Audio Generation with Source Separation and Channel Panning Loss

Spatial audio is essential for many immersive content services; however, it is challenging to obtain or create it. Recently, multimodal-based ambisonic audio generation has emerged as a promising approach for addressing the limitation. It combines multiple modalities, such as audio and video, and provides more intuitive control of ambisonic audio generation. Moreover, it leverages the advantages of machine-learning methods to automatically learn the correlation between different features and generate high-quality ambisonic sounds. Herein, we propose a separation- and localization-based spatial audio generation model. First, the network extracts visual features and separates audio into sound sources. Then, it conducts localization by mapping the separated sound sources to the visual features. To overcome the performance limitation of the previous self-supervised source separation approach, we employ a pretrained source separator with superior performance. To improve the localization performance further, we propose a channel panning loss function between each channel of the ambisonic signal. We use three different types of datasets to train the model experimentally and evaluate the proposed method with four metrics. The results show that the proposed model achieves better spatialization performance than the baseline models.