Distractor Similarity and Item-Stem Structure: Effects on Item Difficulty

This article examined two item-writing guidelines: the format of the item stem and homogeneity of the answer set. Answering the call of Haladyna, Downing, and Rodriguez (2002) for empirical tests of item writing guidelines and extending the work of CitationSmith and Smith (1988) on differential use of item characteristics, a mock multiple-choice driver's license examination was administered to high school students with items having item stems that were either open-ended or in question form and with distractors structured to be either similar or dissimilar to the correct answer. Analyses at the test level indicated that the similarly structured distractors raised the mean difficulty level by .12. No effect was found for item-stem format. Differential item function analyses on each of the test items further supported the effect of distractor similarity on test performance. Implications of this study for item writing and standard setting, as well as implications for future research, are discussed. Notes 1This study was conducted in 1999. 2Because the format of the item stem was also altered across versions, it was sometimes necessary to slightly alter the wording of the options in order to maintain grammatical flow between the item stem and the answer. 3The cyclic counterbalancing procedure maintains the basic sequence of the items but shifts the position to the left for each subsequent sequence. 4The Logistic Regression procedure (c.f. CitationHidalgo & Lopez-Pina, 2004; CitationSwaminathan & Rogers, 1990) and the adjusted MH procedure (c.f. CitationMazor et al., 1994) were also applied. The procedures gave similar results and therefore will not be discussed.