A hybrid audio-visual framework for 3D localization of two active speakers using mediapipe and SALSA-Lite

This study aims to determine the Cartesian coordinates of two active speakers in 3D using data from a webcam and a 4-channel microphone array. The visual processing method involves using the Pose estimation algorithm and Face mesh from the MediaPipe framework. For audio processing, the audio positioning neural network with SALSA-Lite features helps identify acoustic events in elevation and azimuth. Within the fusion phase, the estimations by audio and visual methods are combined using stacking logic. Our investigations found that SALSA-Lite performs poorly in certain conditions such as speaker motion, close proximity, background noise, and spacious areas with reverberations. By processing visual data and fusing the estimated results with SALSA-Lite, the tracking loss rate is reduced from 37.58% in the audio method to 3.37% in the combined audio-visual method. On average, the estimation error (cm) is 8.91 in the audio, 4.6 in the visual, and 4.1 in the combined method.