The Effects of Purification of Matching Criterion on the Identification of DIF Using the Mantel-Haenszel Procedure

To distinguish between differential item functioning. (DIF) and item impact, the Mantel-Haenszel procedure compares the odds for success between groups after conditioning on ability. This conditioning requires a valid criterion, Because an appropriate external criterion is not routinely available, practitioners have typically used the total test score as the matching criterion. Holland and Thayer recommended a potential means of improving that criterion by using a two-step form of the Mantel-Haenszel procedure in which items identified as displaying DIF on an initial implementation are removed from the matching criterion for a subsequent analysis. This study examines the utility of their recommendation by comparing the results of the single-step and two-step procedures in the context of a simulated data set. The results show the two-step procedure to be equal or superior to the single-step procedure in identifying simulated DIF items across all conditions examined. This advantage is achieved without apparent inflation in the Type I error rate. A comparison of the single-step and two-step forms of the procedure is also carried out using an actual data set. The results show changes in identification rate between the two steps of the order of magnitude of those observed with simulated data.