Practical Estimation and Interpretation of Classification Accuracy and Consistency

When the primary test outcome is a proficiency classification, the quality of the classification, rather than the quality of the test score, is of upmost importance. Two widely known indices of classification quality used in mastery testing are classification accuracy (CA) and classification consistency (CC). Recently, there have been advances in item response theory (IRT) methods and accessibility of software that computes these indices; however, practical guidance for supporting interpretation and use of them remains scant. In this study, we aim to diminish this gap between research and practice by providing CA and CC reference values for commonly observed scenarios in mastery testing practice utilizing IRT. We applied Monte Carlo simulation methods to generate data designed to fit the requirements of the Rasch and two-parameter logistic IRT models for 324 test scenarios. We manipulated situational test characteristics familiar to practitioners and known to influence CA and CC, such as test length and number of proficiency levels reported. CA and CC were computed using true and estimated person measures, then averaged across 500 replications. The ways practitioners can use these reference values support the interpretation, evaluation, and use of CA and CC values they observe in their own tests are discussed.