Can LLMs effectively annotate learner writing? Prompting strategies and proficiency effects in annotating L2 learner argumentative essays

Recent research has demonstrated promising performance of large language models (LLMs) in genre-based discourse annotation. However, most existing work has focused on academic research genres, with less attention paid to the annotation of learner writing, where argumentative elements may be realized in diverse and linguistically variable ways. This study investigates LLM-based annotation of argumentative elements in L2 English argumentative essays drawn from the Yonsei English Learner Corpus. The dataset consists of 780 timed argumentative essays (390 beginning-level essays and 390 intermediate-level essays) written by first-year university students. We evaluate model performance under zero-shot, few-shot, and fine-tuning conditions, and analyze residual human–LLM disagreement patterns using human annotations as a reference. Results reveal a clear performance hierarchy, with the fine-tuned model substantially outperforming zero-shot and few-shot prompting conditions in overall annotation performance. However, systematic human–LLM disagreements persist even under the fine-tuning condition, particularly in Claim identification. Qualitative analysis shows that these disagreements are not random inconsistencies but reflect tensions between the argumentative patterns captured by the model and the diverse ways argumentative functions are realized in learner writing. This study advances theoretical understanding of automated discourse analysis in learner corpora and highlights implications for methodological design and pedagogical integration in genre-based writing instruction.