What question did this study set out to answer?

The aim is to improve speech enhancement quality by leveraging transformer-based models for mask estimation.

May 16, 2026CAAI Transactions on Intelligence Technology0 citationsOpen Access

Multiple Transformer‐Based Models for Phase‐Sensitive Masking‐Based Single‐Channel Speech Enhancement

MPMatin PashaianAmirkabir University of Technology SSSanaz SeyedinAmirkabir University of Technology

Key Points

The aim is to improve speech enhancement quality by leveraging transformer-based models for mask estimation.
Proposed and evaluated multiple transformer-based architectures for constrained phase-sensitive ideal magnitude ratio mask (cPSIRM) estimation.
Cascaded structures of CNN and TRF layers (CNN-TRF-cPSIRM) and TRF and CRN layers (TRF-CRN-cPSIRM) developed.
Introduced a transposed transformer-based model (TTRF) to capture both long-term and short-term frequency dependencies.
CNN-TRF-cPSIRM model achieved approximately 0.44 PESQ improvement over the baseline.
The model reached a PESQ score of 1.5 over noisy speech conditions.
Overall transformer-based models improved speech quality and intelligibility compared to existing methods.

Abstract

ABSTRACT Leveraging both global contextual dependencies and local temporal‐spectral patterns can further enhance speech quality and intelligibility, motivating the integration of diverse neural network structures for improved mask estimation. Furthermore, due to the limitations of existing time‐frequency phase‐aware masks, a new constrained phase‐sensitive mask is introduced and estimated using the proposed architectures. In this paper, we first propose a modified transformer model for constrained phase‐sensitive ideal magnitude ratio mask (cPSIRM) estimation (TRF‐cPSIRM) to incorporate both magnitude and phase information and improve speech enhancement quality. The transformer model can extract global information, whereas the convolutional neural network (CNN) and convolutional recurrent network (CRN) are effective in capturing local content. CNNs have feature extraction capability due to their convolutional layers, and CRNs benefit from the temporal modelling strength of LSTM recurrent layers, which is useful for enhancement. Therefore, in this paper, to exploit both transformer (TRF) and CNN/CRN capabilities, we propose the cascaded structures (models) of CNN and TRF layers (CNN‐TRF‐cPSIRM) and also TRF and CRN layers (TRF‐CRN‐cPSIRM) for cPSIRM estimation and, consequently, speech enhancement. The CNN is used for feature extraction in CNN‐TRF‐cPSIRM and CRN for enhancement in TRF‐CRN‐cPSIRM. Moreover, considering the harmonic property of speech in the frequency spectrum, we present the transposed transformer‐based model (TTRF), in which the neighbourhood relationship between the different frequency sub‐bands is used as a sequence in the model. Then, to model both the long‐term and short‐term dependencies, the cascaded model of TTRF (intra‐transformer) and TRF (inter‐transformer) is proposed for cPSIRM estimation (TTRF‐TRF‐cPSIRM). The experimental results show that among these transformer‐based models, the CNN‐TRF‐cPSIRM has the best performance, achieving up to about 0.44 perceptual evaluation of speech quality (PESQ) improvement over the baseline and 1.5 over the noisy speech.

Ask AI

Helpful

Bookmark

View Full Paper