This is the authors' abstract. We don't add key points for this paper.
This research report provides a description of the processes of evaluating the “deployability” of automated scoring (AS) systems from the perspective of large‐scale educational assessments in operational settings. It discusses a comprehensive psychometric evaluation that entails analyses that take into consideration the specific purpose of AS, the test design, the quality of human scores, the data collection design needed to train and evaluate the AS model, and the application of statistics and evaluation criteria. Finally, it notes that an effective evaluation of an AS system requires professional judgment coupled with statistical and psychometric knowledge and understanding of the risk assessment and business metrics.
Answers are grounded in this paper and cite their sources.
No takes yet. Share an insight, caveat, or question.
Rotou et al. (2020) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: