Bayesian Item Fit Analysis for Dichotomous Item Response Theory Models

ABSTRACT Detecting item fit for common dichotomous item response theory (IRT) models has always been an issue of enormous interest, but there exists no unanimously agreed upon item fit diagnostic for the models. This paper employs the posterior predictive model checking method (Guttman, 1967; Rubin, 1981, 1984), a popular Bayesian model checking tool, to examine item fit for common dichotomous item response theory models. An item fit plot, comparing the observed and predicted proportion correct scores of examinee groups (with groups based on raw scores), promises to be useful in real applications. This paper also suggests how to obtain posterior predictive p‐values for the χ 2 ‐type test statistics of Orlando and Thissen (2000) comparing the observed and predicted proportion correct scores for different raw score groups. A number of simulation studies and real data applications examine the effectiveness of the suggested item fit diagnostics. The suggested p‐values seem to have low Type I error rate, low false alarm rate, and adequate power, indicating that researchers might find the methods more acceptable than the existing item fit measures.