Reliability of Auditory-Perceptual Evaluation of Tracheoesophageal Speech: Effects of Listener Number, Listener Subset Variability, and Voice and Speech Severity

Purpose: This study examined how (a) the number of listeners, (b) different subsets of listeners, and (c) the severity of voice/speech samples influence the reliability of auditory-perceptual evaluations of tracheoesophageal (TE) speech for the assessment of overall severity (OS) and listener comfort (LC). Method: Digital audio recordings from 20 male TE speakers were rated by 58 naive listeners for both OS and LC. To assess the impact of the number of listeners and different listener subsets, we gradually increased the number of listeners from five to 40 using repeated random subsampling without replacement and calculated intraclass correlation coefficients (ICCs). Next, an unsupervised clustering approach grouped listeners into “consistent” and “inconsistent” subgroups based on each listener's correlation between OS and LC ratings. Inter- and intrarater reliability were then compared across these subgroups. Finally, we analyzed how the reliability of OS and LC varied across different OS and LC severity ranges based on listener data. Results: As to the number of listeners, reliability increased with more listeners for both OS and LC, with LC achieving strong reliability with fewer raters. Gray-zone samples (mild–moderate severity) continued to yield unstable or even negative ICCs, even with larger listener groups. As to listener subset variability, LC showed tighter bounds and less variability across different listener subsets, whereas OS reliability strongly depended on which raters were included. The clustering method effectively identified inconsistent raters for post hoc screening. As to speech sample severity, both auditory-perceptual scales showed the highest reliability for clearly mild or clearly moderate/severe samples, whereas mild-to-moderate samples produced the lowest and least stable reliability. Conclusions: LC appears to be a robust scale for perceptual evaluation of TE speech, demonstrating higher reliability and requiring fewer listeners than OS. LC also showed substantially less variability across different listener subsets, further supporting its stability and potential clinical utility. Both scales exhibited nonuniform reliability across the severity continuum, highlighting the need for caution when interpreting ratings in the mild-to-moderate range. Enhanced rater training or refined guidelines may improve reliability in this region.