Comparing holistic and analytic scoring methods: issues of validity and reliability

AbstractThis paper explores issues of rating quality when assessing writing in a level-specific approach, i.e. tasks targeting specific proficiency levels. We investigate whether holistic scores could mask deviances in how underlying descriptors are interpreted and applied by the raters, as such deviances could compromise rating validity. We conducted a study with six raters, comparing a holistic approach with a combined approach whereby an analytic score for each descriptor was collected complementary to the holistic judgements. The results confirmed the initial hypothesis that holistic scores may mask deviances in how descriptors are applied. To monitor rating quality and enhance rating validity, we therefore recommend a complementary approach combining holistic scores with analytic, descriptor-focused scores. We illustrate the applicability of this approach by a successful implementation during rater training. Our findings contribute towards enhancing rater consistency and improving rating validity.Keywords: writing assessmentholistic scoringanalytic scoringrating scalesscoring validity AcknowledgementsThe authors would like to thank the raters who participated in this study. We would also like to express our thanks to Ema Ushioda, Keith Richards and the anonymous reviewers for their invaluable feedback during the preparation of this manuscript. The work was carried out at the Institute for Educational Progress (IQB), Humboldt-University Berlin, Germany in collaboration with the IEA-DPC, Hamburg, Germany.Notes1. For an analysis of the effects of task characteristics, performance criteria and raters, see Harsch and Rupp (Citation2011). The fact that each task operationalises one specific CEF-level results in certain restrictions with regard to task expectations and demands, such as structural and linguistic demands, the degree of abstractness or the demanded text type. Tasks targeting a lower level, for example, are usually constrained, demanding short, simple texts which need not – and in some cases cannot – be highly structured or cohesive; the latter characteristics would only be demanded by tasks targeting higher levels. The crucial issue then is to assess performances elicited by level-specific tasks with rating scales which can account for such task demands and constraints. It is a prerequisite for the assessment to represent the level-specific test construct in the rating scale, in analogy to McNamara et al.'s (Citation2002) or Knoch's (Citation2009) rating-validity argumentation. For a level-specific assessment approach, this means that a multi-level rating scale covering all performance levels targeted by the test might not be adequate. For example, the higher levels of such a scale describe features which a performance elicited by a task targeting a lower level cannot expose, such as the 'full and appropriate use of a variety of organisational patterns', as stated in level C1 of the assessment grid in the CEF-Manual (Council of Europe Citation2009, 187).2. Research studies on rater behaviour and rating strategies often employ verbal protocol methods (e.g. Barkaoui Citation2011b; Cumming et al. Citation2002; Lumley Citation2002, Citation2005; Milanovic, Saville, and Shuhong 1996), which, however, have been reported to influence the rating processes and outcomes in varying degrees for different raters (Barkaoui Citation2011b, 68). Hence in contexts where it is the scores and the processes that matter, verbal protocols may introduce a further, unwanted source of variance.3. This analysis is not to be confounded with text analysis, a method which is usually employed in studies focusing on analysing the most salient features for a given assessment, in order to develop a rating scale or investigate output elicited by different prompts (e.g. Cumming et al. Citation2005; Hamp-Lyons Citation1991). In the context we report, the scale was based on existing descriptors, not on text analysis.4. Calculation of the mode is based on all available score options; in our case, it is based on the three scores 'below', 'pass' and 'pass plus', which in turn were coded as 1, 2 and 3 to enable quantitative calculations.5. Since we worked with multiple markings (all six raters scored the same scripts), indices to calculate pair-wise consistency could not be applied. Our data in the feasibility study were too scarce to employ a Rasch-model (see, e.g. Brindley 2000, 75, who regarded his numbers of ratings as too low to justify the use of a Rasch model – he employed 22 spoken and 72 written texts and 12 raters in his study). Note that we only used multiple markings and the index %modagree during the feasibility study and the ensuing rater training. Once raters were trained and showed acceptable agreement levels, and training procedures had confirmed the mode value as representing the 'valid score' in the vast majority of cases, the raters' scores were considered 'valid'. Only then did we start with the 'real' assessment of the scripts collected during the pilot study. In the 'real' assessment, a rating design was used with general single rater markings and a complex multi-matrix design for a sub-sample of multiple markings to monitor (inter- and intra-) rater agreement and consistency. To analyse the scoring quality, multi-faceted Rasch-analysis and G-theory analyses were used. Details on the design and the analyses are reported in Harsch and Rupp (Citation2011).6. Cf. Hayes and Krippendorff (Citation2007) for a very thorough implementation of a chance-corrected reliability index and an explanation of how these differ from 'raw' agreement measures. We opted for using a 'raw' agreement measure because of the difficulties inherent in interpreting pure chance-corrected reliability. This difficulty arises from the fact that chance-corrected reliability, by design, does not necessarily correlate with rating validity. High chance-corrected reliability may not result in valid ratings and valid ratings may not display high chance-corrected reliability. Another reason not to use a chance-corrected reliability index is that this, by design, cannot provide information on the individual rater's performance.