Effectiveness of an Online Perceptual Evaluation of Voice Training Platform on the Rater Reliability in Novice Raters

Purpose: The primary aim of this study was to evaluate the effectiveness of an online perceptual evaluation of voice training platform in improving the rater reliability of novice raters. The secondary aim was to determine the relationship between the training duration and rating accuracy. Method: Forty-two speech-language pathology students participated in this study. Training was conducted using an online training platform over a 12-week period. The online training platform was developed based on perceptual learning theory and includes three main components: (a) familiarization with vocal parameters, (b) training with a single vocal parameter, and (c) training with multiple vocal parameters. Students were instructed to use the training platform to practice auditory–perceptual ratings both inside and outside the classroom. All voice samples included three vocal tasks: sustained vowels, passage reading, and conversational speech. Results: Intrarater exact agreement showed statistically significant improvements after training. Statistically significant increases in rating accuracy were observed for 60% of the vocal parameters between the pretest and the posttest. No significant increases were observed in roughness, breathiness, glottal fry, and pitch breaks after training. There was no correlation between the training duration and the increase in rating accuracy after training. Conclusions: These results indicate that the training platform was effective in improving the rater reliability of novice raters. The training platform employed perceptual learning theory, natural voice samples, an immediate feedback paradigm, and self-paced learning, which can build novice raters' ability to judge voice quality. During training, a longer duration may not provide additional benefits to novice raters. Future research should consider using shorter training durations to train novice raters.