Conservativeness in Rejection of the Null Hypothesis When Using the Continuity Correction in the MH Chi-Square Test in DIF Applications

Conservative bias in rejection of a null hypothesis from using the continuity correction in the Mantel-Haenszel (MH) procedure was examined through simulation in a differential item functioning (DIF) investigation context in which statistical testing uses a prespecified level α for the decision on an item with respect to DIF. The standard MH chi-square test or continuity-corrected MH chi-square test consistently showed a conservative nature across different sample sizes and different significance levels in rejecting the null (No DIF) and non-null (DIF) conditions, especially in small samples. Two alternative testing approaches, the uncorrected MH chi-square and the MH delta-based z test, both showed proper control of Type I error probabilities and better power than the continuity-corrected MH chi-square test. In hypothesis testing for DIF investigations with small sample sizes, either the uncorrected MH chi-square or the MH delta-based z test is recommended rather than the continuity-corrected MH chi-square test.