Computer-based second language production training by using spectrographic representation and HMM-based speech recognition scores

How can we provide feedback to second language (L2) learners about the goodness of their productions in an automatic way? In this paper, we introduce our attempts to provide effective feedback when we train native speakers of Japanese to produce English /r/ and /l/. First, we adopted spectrographic representation overlayed with formant frequencies as feedback. Second, we investigated the correlation between human judgments of L2 production quality and acoustic scores produced by an HMM-based speech recognition system. We also adopted the HMM-based scores as feedback in the production training. Evaluation of the preand post-training productions by human judges showed that production abilities of the trainees improved in both training groups, suggesting that both spectrographic representation and HMM-based scores were useful and meaningful as feedback. These results are discussed in the context of optimizing L2 speech training.