Automated Language Sample Analysis for Children With Developmental Language Disorder

Purpose: Advances in automatic speech recognition (ASR) and natural language processing have the potential to meaningfully impact language assessment. Despite the increasing availability of these technologies, little is known about their performance in routine assessment tasks such as language sample analysis (LSA), which is commonly used in the identification of developmental language disorder (DLD). This study examined the performance of ASR in transcribing child language samples, as well as the concurrent validity and diagnostic accuracy of metrics derived from an automated LSA system. Method: We used an existing corpus of narrative samples from 66 children with DLD and 285 with typical language (TL), which included audio recordings and transcripts. We conducted automated LSA using the proprietary software LSA+, which integrates ASR and natural language processing. We evaluated transcription accuracy by comparing ASR transcripts to those included in the corpus. We then examined associations between language ability metrics from ASR and manual transcripts, as well as the diagnostic accuracy of each, by using LSA+ as a common means of analysis for all transcripts. Results: ASR produced less accurate transcripts for children with DLD than for those with TL and for younger children. Despite differences in transcription accuracy, automated LSA measures exhibited positive correlations with their manual counterparts, ranging from r = .79 to .98. Manual LSA measures in isolation exhibited better diagnostic accuracy than that of their automated counterparts. However, composite models with multiple automated metrics yielded the highest accuracy when including mean length of utterance, number of different words, and total number of words, in addition to ASR-derived mean length of speech segment and pauses per word. Discussion: Full automation of LSA demonstrates the potential to improve the efficiency and robustness of language assessment, particularly when utilizing composite measures. Despite its advantages, automated LSA may result in inaccurate estimates of language ability if used without consideration of its limitations. Continued examination of ASR and related technologies is needed to better understand their efficacy across languages, ages, and types of communication disorders.