Expert Judgments as an Approach to Evaluating Items’ Instructional Sensitivity

Instructional sensitivity is an item characteristic that refers to the extent to which an item captures the effects of instruction on student performance. One potentially economical approach to evaluate instructional sensitivity is based on expert judgments. However, little is known about their validity. This study investigates expert judgments from four perspectives: a comparison (1) of teachers and subject matter specialists; (2) of a global and a differentiated rating; (3) of the differentiated rating facets, and (4) of expert judgments to statistical measures of sensitivity. Results indicate that (a) teachers perform better than subject matter specialists, (b) the differentiated rating offers no advantage over the global rating, (c) experts can distinguish the facets of the differentiated rating, and (d) expert judgments and item sensitivity measures are weakly to moderately related. We conclude that expert judgments show potential but are not (yet) sufficiently accurate to replace established procedures for investigating instructional sensitivity.