ABSTRACT This report evaluates the construct coverage of the e‐rater® scoring engine. The matter of construct coverage depends on whether one defines writing skill , in terms of process or product. Originally, the e‐rater engine consisted of a large set of components with a proven ability to predict human holistic scores. By organizing these capabilities into features, e‐rater researchers organized the e‐rater engine along the lines of trait scoring, which recognizes that essay quality has several dimensions. Some traits of essay quality cut across different methods for scoring essay quality, such as the rubrics employed by the GRE® and TOEFL® assessments, as well as the 6‐trait scoring model. Factor analyses conducted by Attali and Powers (2008) suggest that e‐rater features capture low‐level aspects of essay quality, such as sentence complexity, vocabulary, and conventions. Future e‐rater development should focus on (a) deepening and expanding coverage of the construct, such as by developing measures of essay content and organization, as well as on (b) addressing accuracy issues in existing features.
Quinlan et al. (Mon,) studied this question.