Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

In this article, procedures are described for estimating single‐administration classification consistency and accuracy indices for complex assessments using item response theory (IRT). This IRT approach was applied to real test data comprising dichotomous and polytomous items. Several different IRT model combinations were considered. Comparisons were also made between the IRT approach and two non‐IRT approaches including the Livingston‐Lewis and compound multinomial procedures. Results for various IRT model combinations were not substantially different. The estimated classification consistency and accuracy indices for the non‐IRT procedures were almost always lower than those for the IRT procedures.