Using machine learning to identify causes of differential item functioning

Assessment programs frequently assess for evidence of differential item functioning in order to support the fairness and equity of assessments; however, attempting to understand why differential item functioning is occurring is much less prevalent. In this study, a machine learning framework was proposed to identify factors contributing to differential item functioning, and a simulation study and empirical data analysis were conducted to demonstrate the feasibility of the proposed framework. Log-linear cognitive diagnosis models were used to model student responses to assessment items, and the logistic regression differential item functioning detection method was used to flag items for evidence of differential item functioning. Logistic regression, k-nearest neighbors, random forest, naïve Bayes, support vector machines, and artificial neural network models were estimated to predict item responses for items flagged for evidence of differential item functioning using estimated attribute mastery status, gender, and ecological variables as predictors. The logistic regression, k-nearest neighbors, random forest, and naïve Bayes models were found to be feasible, with the logistic regression, random forest, and naïve Bayes models being able to be estimated in 100% of repetitions and k-nearest neighbors being able to be estimated in greater than 99% of repetitions. The computational requirements for the support vector machines and artificial neural networks rendered these models infeasible, given the time required to estimate these models. The average classification accuracy of the logistic regression and random forest models was approximately .70, with the k-nearest neighbors and naïve Bayes models performing slightly poorer. The average area under the receiver operating characteristic curve for the logistic regression, naïve Bayes, and random forest models was approximately .70, with the k-nearest neighbors model performing slightly poorer. While the logistic regression, k-nearest neighbors, random forest, and naïve Bayes models performed at rates above chance, the performance of these models was suboptimal for operational use. Additional research is needed to refine the proposed machine learning framework for operational use.