Key points are not available for this paper at this time.
Various factors contribute to variability in English as a second language (ESL) essay scores and rating processes. Most previous research, however, has focused on score variability in relation to task, rater, and essay characteristics. A few studies have examined variability in essay rating processes. The current study used think-aloud protocols to examine the roles of rating scales, rater experience, and interactions between them in variability in raters' decision-making processes and the aspects of writing they attend to when reading and rating ESL essays. The study included 11 novice and 14 experienced raters, who each rated 12 ESL essays, both holistically and analytically, while thinking aloud. The findings indicated that rating scale type had larger effects on the participants' rating processes than did rater experience. With holistic scoring, raters tended to refer more often to the essay (the focus of the assessment), whereas with analytic scoring they tended to refer to the rating scale (the source of evaluation criteria) more frequently; analytic scoring drew raters' attention to all evaluation criteria in the rating scale, and novices were influenced by variation in rating scales more than were the experienced raters. The article concludes with implications for essay rating practices and research.
Khaled Barkaoui (Fri,) studied this question.