Sequential Speaker Diarization with Dual-Model Identifier Buffers and Overlap-Aware Assignment

This paper presents a sequential speaker diarization framework designed to improve robustness and computational efficiency in multi-speaker recordings containing overlapping speech, varying speaker counts, and diverse acoustic conditions. Unlike conventional clustering-based approaches, which often suffer from error propagation and limited overlap handling, and end-to-end neural systems that typically impose fixed speaker capacity constraints, the proposed framework combines adaptive speaker assignment strategies with dual speaker embedding representations to achieve stable speaker tracking while remaining sensitive to previously unseen speakers. The system further incorporates an explicit overlap-handling mechanism based on softmax-weighted cosine-similarity aggregation over previously assigned speaker embeddings. The proposed framework is compatible with multiple segmentation front-ends, including Pyannote and DiariZen, and is evaluated on eight benchmark datasets: VoxConverse, AMI-IHM, AMI-SDM, CallHome, MSP-Podcast, AISHELL-4, AliMeeting, and MSDWild. Experimental results demonstrate diarization error rates competitive with state-of-the-art systems across most datasets while significantly reducing inference latency, achieving processing speeds 2.5 × faster than standalone DiariZen and 1.6 × faster than Pyannote. In addition, the proposed system outperforms Ultra-SortFormer on most of the evaluated datasets.