Accuracy of Bias Review Judges in Identifying Differential Item Functioning on Teacher Certification Tests

The purpose of this study was to examine whether or not judges on bias review committees can identify test items that function differently for Black and White examinees. Judges (N = 42) on three bias review committees were asked to examine a set of items, and predict differential item functioning without empirical data. Test items from teacher certification tests in the content fields of Early Childhood, Administration and Supervision, and Middle Childhood were examined here. Each committee examined 40 items, and agreement between judgmental and empirical indices of differential item functioning were determined. The results of this study suggest that the agreement between the bias review judges and the empirical indices are generally not beyond what would be expected by chance, although each field had one to two judges who exhibited statistically significant agreement with the empirical indices of differential item functioning. Suggestions for future research on identification of biased items and the practical implications of this study are discussed.