Using Topic Models to Understand Rater-Mediated Writing Assessments

Integrated writing assessments are tasks that require students to construct an essay using evidence from source texts. Topic models are statistical models used to analyze text. This study analyzes two integrated writing tasks from students in grades 6 (N=1,250) and 8 (N=1,250) using topic models. Independent evaluations of the essays were also conducted to help generalize the results across the two grades and better understand students' underlying writing techniques. Four main categories were identified: (1) use of words that appeared in the prompt, (2) words that appeared in the source texts, (3) words that reflected inference, and (4) words that reflected integration. Higher-scoring students used words that reflect inference and integration of source text more often than lower-performing students.