Target-speaker automatic speech recognition with dual-mode training

Target-speaker automatic speech recognition (TS-ASR) is a critical technology for transcribing a specific speaker's voice in multi-talker environments. By providing target speaker clues, it can also improve performance in noisy single-speaker situations. Conventionally, TS-ASR models are designed for either high-accuracy offline (batch) processing or low-latency real-time (streaming) inference, requiring separate models for different applications. This approach is inefficient, increasing development and maintenance overhead. This paper introduces a dual-mode training framework to TS-ASR that enables a single model to operate effectively in both streaming and batch modes. During training, both modes are executed simultaneously, compelling the model to learn robust representations that are effective with limited and complete future context. This approach potentially enriches the streaming mode's performance. We explore both a cascaded model that integrates target speaker extraction (TSE) with ASR and an end-to-end system. The proposed framework is evaluated using the LibriMix benchmark, demonstrating its capability to create a flexible and resource-efficient TS-ASR system.