Answers are grounded in this paper and cite their sources.
Demonstrates category-specific severity shifts in rater assessments, indicating significant implications for scoring accuracy.
The many‐facet Rasch model (MFRM), widely used to analyze and evaluate rater‐mediated assessments, focuses on between‐rater differences in overall severity or leniency across facets and rating scale categories. Studying within‐rater severity differences, particularly abrupt shifts in a rater's tendency to assign harsh or lenient ratings around specific scale categories, requires an extended modeling approach. Specifically, to focus on local, category‐dependent severity effects, we propose the “many‐facet Rasch model for category‐specific severity shifts” (MFRM‐CSS). Building on Bayesian parameter estimation, we demonstrate the model's suitability for examining local severity changes at scale categories with special significance as cut scores. In a simulation study and real‐data analysis, we found that (a) differences in category‐dependent severity levels affected observed score distributions and passing rates, (b) the MFRM‐CSS reliably recovered true overall and local rater severity parameters, (c) ignoring local severity shifts biased overall severity and category threshold estimates, and (d) the MFRM‐CSS outperformed the baseline MFRM, which does not account for category‐specific severity shifts, in data‐model fit when applied to essay rating data. In the real dataset, the greatest impact of local severity was observed at the second threshold, where the pass‐fail cut score was set. The discussion highlights the practical implications of applying the MFRM‐CSS and suggests directions for future research.
No takes yet. Share an insight, caveat, or question.
Jin et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: