Answers are grounded in this paper and cite their sources.
Scores from formal classroom observation systems are used for many purposes in both research and practice. One important question arising from these uses concerns when observation scores should be adjusted for student characteristics, as scores are often highly correlated with student characteristics. This paper considers this question from a measurement perspective, highlighting three basic interpretations of scores and five pathways through which scores could correlate with student characteristics. Adjusting observation scores is only recommended when interpreting scores as representing teacher quality and for some pathways, which implies that complex, partial adjustments of the relationship between observation scores and student characteristics are likely necessary. Given the difficulty of such partial adjustments, the paper presents bounding exercises to explore the potential size of the bias introduced to teacher quality estimates. The results suggest that rater error threatens score interpretations more than whether scores are adjusted for student characteristics.
No takes yet. Share an insight, caveat, or question.
Mark White (2024) studied this question.