Assessing intelligibility as conceptualised within the CEFR-companion volume (CV) framework using Adaptive Comparative Judgement

Two pivotal constructs, intelligibility (listeners’ actual understanding) and comprehensibility (listeners’ perceived difficulty of understanding), have dominated second language (L2) pronunciation research, marking a shift away from an emphasis on nativeness. The 2020 Companion Volume to the Common European Framework of Reference for Languages (CEFR-CV) presented a revised phonological competence scale, integrating both dimensions into a new definition of intelligibility. However, effective measurements to assess this refined construct are still lacking. This study explores the potential of Adaptive Comparative Judgement (ACJ) in measuring intelligibility as conceptualised by the CEFR-CV. ACJ employs judges who evaluate two stimuli based on a holistic criterion, selecting the better one. Through a collection of such binary decisions, judges’ evaluations are statistically analysed, producing standardised estimates for each stimulus. Twelve judges assessed speech samples in English produced by 30 Mandarin first language speakers on four sentence repetition tasks. Incorporating Think Aloud Protocol (TAP) into the judgement process, the study combined quantitative and qualitative analyses, providing evidence for the efficacy of ACJ in measuring L2 speech. The results, discussed in reference to existing literature on intelligibility and comprehensibility, reveal potential future research pathways on the use of ACJ in L2 pronunciation assessment to further elucidate CEFR-CV’s intelligibility construct.