Second Language Interactional Competence in Roleplay: Comparing Generative Artificial Intelligence and Human Interlocutors

Abstract This study explores the potential of generative artificial intelligence (GenAI) as an alternative to human interlocutors for assessing interactional competence (IC) in a second language (L2). Thirty L2 English speakers completed a 6‐item roleplay task designed to elicit refusals of requests, invitations, and offers, interacting with both a native‐speaking human interlocutor and GenAI (ChatGPT‐4o) 1 week apart. Four trained raters evaluated the audio‐recorded performances over a 2‐week period following an analytic, data‐driven rubric of IC comprising eight dimensions. Linear mixed‐effects models were used to compare IC scores across the two interlocutor conditions. Rater feedback collected through daily questionnaires was analyzed to contextualize rating outcomes. Results indicate that GenAI can serve as a viable alternative to human interlocutors for eliciting IC evidence on the sequential organization of refusals as disaffiliative deviations from acceptance in response to invitations, offers, and requests, though human interlocutors elicited richer IC features indicative of conversational engagement.