Answers are grounded in this paper and cite their sources.
We discuss a new dimension of quantitative validity evidence that considers subjective rater uncertainty in assigning item scores via the use of rater-elicited confidence scores and the mathematical apparatus of random-variable-valued measurements. We examine implications for inter-rater reliability and construct validity for 16 raters who assessed fundamental motor skills using an augmented version of the Test of Gross Motor Development-3rd Edition to capture their subjective confidence in assigning definitive binary scores for each item. Analysis of the augmented test revealed substantial subjective rater uncertainty in 9 of 50 components, considerable variation in subjective uncertainty across equally trained raters, and a tendency to produce higher total and subscale scores when using only a traditional binary scoring system. These findings suggest that there are important subjective differences in how different raters interpret and apply common criteria for assessing fundamental motor skills and highlight the need for centralized training of raters before clinical application.
No takes yet. Share an insight, caveat, or question.
Kroc et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: