Measurement Characteristics of Content-Parallel Single-Best-Answer and Extended-Matching Questions in Relation to Number and Source of Options

BACKGROUND: Previous research showed that extended-matching questions (EMQs) with eight options per set resulted in better score precision than EMQs with larger numbers of options or independent single-best-answer items (A-type) with five options. This study extends previous work using smaller numbers of options. METHOD: Ninety-six questions were presented in two formats on United States Medical Licensing Examination Step 2: as two-item EMQ sets and as independent A-types. Four versions of EMQs were used: five- and eight-option versions with options selected using statistics, and five- and eight-option versions with options selected by physicians. Seven A-type versions were used: three-, four-, five-, and eight-option versions with options selected using statistics, and three-, four-, and five-option versions with options selected by physicians. RESULTS: Items with more options were harder, required more time to complete, and had similar item discrimination. Option sets selected by physicians were easier, slightly more discriminating, and required less testing time. CONCLUSIONS: A-types with four or five options and EMQs with eight options make more efficient use of testing time. Provision of response statistics to content experts does not seem necessary to guide option selection.