Enhancing HuBERT for spontaneous speech with acoustic feature fusion

Spontaneous and semi-spontaneous speech pose challenges for automatic speech recognition (ASR) systems due to pronunciation variation, reduction, and disfluencies. This study aims to explore whether the performance of a pre-trained speech model on this type of speech can be improved with the addition of some acoustic-prosodic information. Using the openSMILE toolkit, 10 low-level acoustic-prosodic features are extracted from the eGeMAPSv02 set, including loudness, F0, jitter, shimmer, and the frequencies and bandwidths of the first three formants (F1, F2, and F3). We fine-tuned the HuBERT model (Hubert-large-ls960-ft) on a subset of the GigaSpeech corpus consisting of YouTube and podcast recordings. To align the acoustic features with HuBERT’s frame resolution, the features are normalized, interpolated and projected through a linear projection layer. The projected features are subsequently fused with HuBERT’s hidden representations using a gated fusion mechanism. Preliminary results show that the feature-fusion model outperforms the baseline HuBERT model, particularly in recognizing spontaneous or semi-spontaneous speech. Moreover, the study aims to identify the most informative features among the 88 available in the eGeMAPSv02 set. By assessing each feature's impact on spontaneous speech recognition, the study also investigates the broader applicability of the proposed approach via publicly available datasets.