DaNangVMD: Vietnamese Speech Mispronunciation Detection

Automatic Speech Recognition, also known as ASR, has grownexponentially over the past decade and is used to recognize and translate humanspeech into readable text automatically. However, Vietnamese SpeechRecognition faces critical challenges such as easy to make mispronunciations aswell as a huge variant in Vietnamese speech. In this work, we dive into thedifficult challenge of Mispronunciation Detection (MD) in the Vietnameselanguage. As such a tonal language, Vietnamese is not only based on consonantsand vowels but also on variations in pitch or tone during pronunciation. In thispaper, we propose DaNangVMD model for detecting mispronunciations inVietnamese speech based on the audio speech and canonical transcript. Byleveraging multi-head attention-based multimodal representation from theembeddings of the phonetic encoder and linguistic encoder, DaNangVMD aimsto provide a robust solution for accurate mispronunciation detection anddiagnosis. Throughout the extensive evaluation, the proposed DaNangVMDexhibits superior performances rather than that of the PAPL baseline models by15% in F1 score and 13 % in accuracy.