How Consistent Are Modern DIF Methods? Evidence From an Operational General Education Test

Differential item functioning (DIF) analyses contribute evidence relevant to fairness, yet limited empirical work has directly compared newer DIF procedures on operational testing data. This study contrasted three modern DIF methods, Weighted Sum Score (WSS), Robust DIF, and Residual DIF (RDIF), using dichotomously scored data from first-year undergraduates (n = 3,630) who completed a 33-item general education assessment (GLEX4). DIF by sex was evaluated after examining the suitability of a unidimensional 2PL model through item fit, local independence, and unidimensionality checks; two clearly misfitting items were removed prior to DIF analyses. WSS (uniform DIF only), Robust DIF (robust scaling in a common-item design), and RDIF (residual-based indices from pooled calibration) were implemented with false discovery rate control. All three procedures flagged a meaningful set of items for DIF but did not produce identical item classifications. Agreement was strongest for uniform DIF, with the methods converging on a shared subset of items reflecting group differences in difficulty. In contrast, divergence was more pronounced for nonuniform DIF, where RDIF identified more slope-related differences than Robust DIF and WSS was not applicable by design. Overall, findings indicate that these procedures yield partially overlapping DIF profiles and should be treated as complementary rather than interchangeable, with statistical flags interpreted alongside structured substantive item review in applied fairness evaluations.