Differential Accuracy of Semantic Versus Phonological Speech Error Transcription in Aphasia: Comparing Large Language Model (Whisper) to Clinician (Speech-Language Pathologist) Performance

Purpose: Patient speech samples provide insights into cognitive and linguistic status, yet their utilization has been limited by labor-intensive manual transcription. Advances in automated speech recognition (ASR) with large language models (LLMs) potentially save time, but performance on impaired speech, such as due to poststroke aphasia, remains understudied. Method: We transcribed 4,535 trials of the Philadelphia Naming Test (PNT) administered to 26 people with aphasia (PWA) using OpenAI's Whisper speech-to-text LLM. These ASR transcripts were compared with gold-standard transcriptions by an experienced speech-language pathologist (SLP). Clinician–LLM match or mismatch in transcription was analyzed across speech-error types (semantic vs. phonemic), patient severities, and word frequency. Results: For each 175-trial PNT, transcriptions by SLP required 180 ± 40 min versus 2.3 ± 0.5 min by Whisper. Compared to > 95% match rate with SLP transcripts for trained humans, Whisper achieved an overall match rate of 75.52%, subdivided as 88.83% match on correct PNT trials, 92.22% on semantic errors, and 23.05% for trials with phonemic errors by PWA. Western Aphasia Battery–Aphasia Quotient scores correlated with match rate ( R 2 = .196, p = .023) and hallucination rate ( R 2 = .291, p = .0048). Low-frequency words showed higher mismatch than medium- or high-frequency words (analysis of variance F = 14.26, p < .001). Conclusions: Whisper reliably transcribed semantic but not phonological errors, suggesting automation is best for fluent aphasia types with less phonemic distortions. This study benchmarks LLM transcription for impaired speech, offering a scalable alternative to manual transcription that may streamline clinical and research workflows. Supplemental Material: https://doi.org/10.23641/asha.34790292