Lightweight Multi-Window U-Net for Data-Efficient Frame-Level Speaker Segmentation

Speaker segmentation identifies the temporal regions in which speakers are active and is a fundamental component of speaker diarization, meeting transcription, conversational analytics, and overlap-aware speech processing. Conventional U-Net architectures are designed for two-dimensional image segmentation. It is computationally inefficient for speech representations when only the temporal resolution needs to be recovered. We have adopted a novel U-Net with reduced expansion in the frequency axis at the decoder, called asymmetric U-Net. This can reduce the parameters because only the time intervals are required to be segmented. Another challenge in speaker segmentation is handling varying lengths of different utterances. We propose a modified U-Net by imposing varying receptive fields on convolution kernels to effectively find multi-scale speaker activities. The proposed model contains 2.42 million parameters, compared with approximately 31.04 million parameters for the conventional U-Net baselines. We have carried out experiments independently on mixtures derived from LibriSpeech and WSJ0. The proposed model achieved the best Libri-mix frame accuracy at every scale and showed particularly strong low-data performance on WSJ0-mix. Results with small training sets indicate that speech-specific asymmetric restoration and multi-scale local context can substantially reduce model size and improve frame-level segmentation accuracy, especially when labeled mixtures are scarce.