Analysis of Deepspeech2 Transfer Learning for Balinese-Accented English Speech Recognition

Automatic speech recognition (ASR) systems are generally trained on speech corpora dominated by native speakers, resulting in higher recognition error rates for speakers with underrepresented accents. This study investigates the effect of the Balinese accent on DeepSpeech2 performance and evaluates the effectiveness of transfer learning for model adaptation. A dataset comprising 161 English utterances spoken by Balinese-accented speakers was collected using reading materials derived from LibriSpeech transcripts. The same textual content was maintained across native and Balinese-accented speaker evaluation conditions to minimize textual variation and isolate the influence of accent on recognition performance. The results indicate that the Word Error Rate (WER) increased from 43.2% for native speakers to 85.3% for Balinese-accented speakers, demonstrating a substantial performance gap. Transfer learning using a conventional layer-freezing approach failed to improve recognition accuracy and instead degraded performance across both speech domains. Further investigation revealed that this degradation was associated with shifts in Batch Normalization statistics, which continued to be updated during training despite the affine parameters being frozen via the requires_grad mechanism. Since these statistics are stored as buffers rather than trainable parameters, conventional parameter freezing does not prevent their modification during forward passes. To address this issue, Batch Normalization statistics were explicitly frozen by switching the corresponding layers to evaluation mode using eval(). With this correction, transfer learning reduced the WER for Balinese-accented speech to 81.3%, while limiting performance degradation on the source domain to 3.3 percentage points. The corrected freezing strategy consistently outperformed conventional freezing across all tested configurations. These findings demonstrate that proper management of Batch Normalization statistics is essential for effective transfer learning and highlight a general consideration for adapting neural networks through layer freezing