An Examination of the Effect of Multiple-Choice Item Writing Quality on Test Performance: Differential Effects on Subgroups?

The multiple-choice test format is arguably the primary method in assessments of knowledge and other cognitively loaded constructs associated with subgroup differences. Although widely accepted item writing guidelines exist, seemingly no empirical studies have examined the extent to which violations of these guidelines result in differential test scores particularly as a function of race/ethnicity – despite assertions to that effect. This gap is addressed by (a) examining the extent to which poor item writing practices affect test scores, and (b) whether said effect is moderated by test taker race/ethnicity, using a correlational study with operational police selection data (N = 60,936), and an experimental study with undergraduate students (N = 612). Although the effects were generally small, test takers consistently performed worse on "bad"/poor quality-item exams. However, although generally in the expected direction, the results failed to support a statistically significant case that bad items' deleterious effect varied by race/ethnicity. Nevertheless, because the results likely represent a lower bound of the potential impact of item quality due to an imperative methodological design decision that precluded the inclusion of common egregious rule violations (e.g., direct clue errors, double negations), they highlight the importance of adherence to scientific and professionally recognized item writing guidelines.