End-To-End Speaker Segmentation for Overlap-Aware Resegmentation

Speaker segmentation consists in partitioning a conversation between one or speakers into speaker turns. Usually addressed as the late combination of sub-tasks (voice activity detection, speaker change detection, and speech detection), we propose to train an end-to-end segmentation that does it directly. Inspired by the original end-to-end neural speaker approach (EEND), the task is modeled as a multi-label problem using permutation-invariant training. The main is that our model operates on short audio chunks (5 seconds) but at much higher temporal resolution (every 16ms). Experiments on multiple speaker datasets conclude that our model can be used with great success on voice activity detection and overlapped speech detection. Our proposed can also be used as a post-processing step, to detect and correctly overlapped speech regions. Relative diarization error rate improvement the best considered baseline (VBx) reaches 17% on AMI, 13% on DIHARD 3, 13% on VoxConverse.