This report presents a systematic investigation into fine-tuning OpenAI's Whisper-small automatic speech recognition (ASR) model for the UXSSD (Ultrax Speech Sound Disorders) subset of the UltraSuite repository. This dataset contains real ultrasound and acoustic speech data from children with speech sound disorders collected during speech therapy sessions. The work is conducted as part of developing an AI-driven assistive application for early childhood education called "Kidzinya," which integrates multiple AI features including speech-to-text for pronunciation practice. This study focuses specifically on the speech-to-text component, documenting five iterative training experiments to optimize the Whisper-small model for child speech transcription. The Whisper-small model (244M parameters) was selected based on the need for on-device or edge deployment feasibility while maintaining accuracy for child speech. Child speech differs significantly from adult speech in acoustic and linguistic patterns, presenting unique challenges for ASR systems. Key methodologies employed include the WhisperProcessor for audio feature extraction and tokenization, a custom DataCollator for batch processing, and the Seq2SeqTrainer for automated training loops. Word Error Rate (WER) and Character Error Rate (CER) served as evaluation metrics. Using precomputed features loaded from pickle files (2,800 training samples, 350 validation samples, 350 test samples), the study executed five experimental runs with progressively optimized hyperparameters. Through iterative tuning of learning rate, weight decay, warmup steps, and gradient accumulation, the final model configuration achieved a validation WER of 13.71% and CER of 9.38%. This represents a significant improvement from initial attempts that exhibited overfitting and higher error rates. This work was conducted at Arab International University (AIU), Syria. The official website of the university is: https://www.aiu.edu.sy
Naya Sakka (Sat,) studied this question.