An Investigation of Classification Consistency Indexes Estimated Under Alternative Strong True Score Models

Using several data sets, the authors examine the relative performance of the beta binomial model and two other more general strong true score models in estimating several indexes of classification consistency. It is shown that the beta binomial model can provide inadequate fits to raw score distributions compared to more general models. This lack of fit is reflected in differences in decision consistency indexes computed using the beta binomial model and the other models. It is recommended that the adequacy of a model in fitting the data be assessed before the model is used to estimate decision consistency indexes. When the beta binomial model does not fit the data, the more general models discussed here may provide an adequate fit and, in such cases, would be more appropriate for computing decision consistency indexes.