Dif Analysis for Pretest Items in Computer‐adaptive Testing

ABSTRACT A simulation study of methods of assessing differential item functioning (DIF) in computer‐adaptive tests (CATs) was conducted by Zwick, Thayer and Wingersky (in press; 1993). Results showed that modified versions of the Mantel‐Haenszel and standardization methods work well with CAT data. DIF methods were also investigated for nonadaptive “pretest” items, for which item parameter estimates were assumed unavailable. The pretest DIF statistics were generally well‐behaved, but the Mantel‐Haenszel DIF statistics tended to have larger standard errors for the pretest items than for the CAT items. The current extension of the earlier work addressed the effect of using alternative matching methods for pretest items. Using a more elegant matching procedure did not lead to a reduction of the Mantel‐Haenszel standard errors and produced DIF measures that were nearly identical to those from the earlier study. Further investigation showed that the Mantel‐Haenszel standard errors tended to be larger when items were administered to examinees with a wide ability range, whereas the opposite was true of the standard errors of the standardization DIF statistic. Some theoretical findings were obtained that appear to explain this phenomenon.