Multi-talker conversational speech generation for training speaker diarization model via text-to-speech

Speaker diarization is a task that estimates “who spoke when” from multi-talker audio. Recent advancements in end-to-end models have significantly improved diarization performance, especially for overlapping speech. However, these models require a large amount of annotated multi-talker audio data for training. Such annotated datasets are limited, and manual annotation is highly costly. Therefore, previous studies generate training data by artificially concatenating segments from recorded audio. Nevertheless, these methods largely rely on recorded audio, with little consideration given to the content of the spoken utterances. The utterances are often contextually unrelated. In this work, we propose a data generation method using text-to-speech (TTS) synthesis, which enables the preparation of speech from arbitrary text. Leveraging TTS enables the generation of audio with flexible utterance lengths and speech quantity. Furthermore, by using multi-speaker conversational text, it becomes possible to generate multi-talker conversational speech that preserves context across utterances. Through experimental evaluations, we will investigate how training audio duration, per-speaker speech quantity, and contextual coherence impact diarization performance. We aim to show that TTS-based data generation can effectively improve speaker diarization performance.