Among different factors affecting writing assessment, rater and rating scale are two influential variables that determine the outcome of assessment. In fact, the way raters interact with rating scales considerably affects the validity of their assessment. Taking this into account, this study attempted to identify and classify the raters’ behaviors in the Iranian EFL context when using analytic and holistic rating scales. To this end, a group of nine expert raters were asked to verbalize their thoughts when rating student essays. They were also asked to do their rating using the analytic scale of ESL Composition Profile and IELTS holistic scale. Upon the qualitative analysis of think-aloud protocols (TAPs), two themes emerged which showed the raters’ behaviors when applying the rating scales. The findings further showed that when using the holistic scale, the raters read the text first to get an overall impression. Then they assessed the text based on their own criteria. Next, they referred to the scale for scoring and in the last stage they provided evidence for their scoring. On the other hand, when applying analytic rating scales, the raters first scanned the text for surface features. Then they read the text for their initial impression. Next, they read each scale component and its descriptor for scoring and finally, they attempted to provide evidence for their scoring. In addition to identifying the raters’ behaviors, the raters’ behaviors were classified to shed light on the process of rating. The findings imply that the diagnosis of the rater-rating scale interactions can unveil the strengths and weaknesses of the EFL rating process. This, in turn, can provide more quality training for the raters and in this way enhance the scoring validity of their judgements. In the long run, this would improve the professional development of the raters involved in the writing assessment.
Khodaparast et al. (Sat,) studied this question.