Answers are grounded in this paper and cite their sources.
Researchers of high-stakes, subjectively scored writing assessments have done much work to better understand the process that raters go through in applying a rating scale to a language performance to arrive at a score. However, there is still unexplained, systematic variability in rater scoring that resists rater training (see Hoyt & Kerns, 1999 Hoyt, W. T. and Kerns, M. D. 1999. Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4: 403–424. [Crossref], [Web of Science ®] , [Google Scholar]; McNamara, 1996 McNamara, T. 1996. Measuring second language performance, New York, NY: Longman. [Google Scholar]; Weigle, 2002 Weigle, S. C. 2002. Assessing writing, Cambridge, UK: Cambridge University Press. [Crossref] , [Google Scholar]; Weir, 2005 Weir, C. J. 2005. Language testing and validation: An evidence-based approach, New York, NY: Palgrave MacMillan. [Crossref] , [Google Scholar]). The consideration of individual differences in rater cognition may explain some of this rater variability. This mixed-method exploratory case study (Yin, 2009 Yin, R. K. 2009. “Case study research: Design and methods”. Thousand Oaks, CA: Sage. [Google Scholar]) examined rater decision making in a high-stakes writing assessment for preservice teachers in Quebec, Canada, focussing on individual differences in decision-making style, or “stylistic differences in cognitive style that could affect decision-making” (Thunholm, 2004 Thunholm, P. 2004. Decision-making style: Habit, style, or both?. Personality and Individual Differences, 36: 931–944. [Crossref], [Web of Science ®] , [Google Scholar], p. 932). The General Decision Making Style Inventory questionnaire (Scott & Bruce, 1995 Scott, S. G. and Bruce, R. A. 1995. Decision-making style: The development and assessment of a new measure. Educational and Psychological Measurement, 55: 818–831. [Crossref], [Web of Science ®] , [Google Scholar]) was administered to six raters of a high-stakes writing exam in Quebec, and information on the following rater behaviours was also collected for their potential for providing additional information on individual decision-making style (DMS): (a) the frequency at which a rater decides to defer his or her score, (b) the underuse of failing score levels, and (c) the comments provided by raters during the exam rating about their decisions (collected through “write-aloud” protocols; Gigerenzer & Hoffrage, 1995 Gigerenzer, G. and Hoffrage, U. 1995. How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102: 684–704. [Crossref], [Web of Science ®] , [Google Scholar]). The relative merits of each of these sources of data are discussed in terms of their potential for tapping into the construct of rater DMS. Although score effects of DMS have yet to be established, it is concluded that despite the exploratory nature of this study, there is potential for the consideration of individual sociocognitive differences in accounting for some rater variability in scoring.
No takes yet. Share an insight, caveat, or question.
Beverly Baker (2012) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: