Single-channel speech source separation for speech recognition and speaker verification

A method for single-channel separation of two speech sources based on conditional generative modeling is proposed. The task is characterized by uncertainty in the order of the two reconstructed speech signals: the same pair of separated sources can be represented in two permutations. To resolve this ambiguity, a frozen speaker encoder is used: it defines the source order during training, selects the best variant among N stochastic generations, and aligns the sources during chunk-wise processing of recordings of arbitrary length. On the Libri2Mix dataset, the proposed Transformer U-Netbased variant is competitive in terms of fundamental metrics for evaluating the quality and intelligibility of processed speech and achieves the best results when used in speech recognition and speaker verification tasks.