Classification Accuracy of Assigning Student Performance to Proficiency Levels: Guidelines for Assessment Design

In this study we examined the degree of variation in the accuracy of classifying student performance to proficiency-level scores with changes in the number of proficiency levels and the measurement accuracy. Furthermore, we examined the degree to which the classification accuracy varies across different ability levels given different numbers of proficiency levels based on the same test and the same set of cut-scores. The results of the study based on simulations indicate that the classification accuracy decreased, on average, by 10% for an increase of 1 proficiency level, 20% for an increase of 2 proficiency levels, and 20% to 30% for an increase of 3 proficiency levels. In addition, classification accuracy varied 10% to 20% for tests with reliabilities that ranged between 0.70 and 0.93. The findings regarding the variability of classification accuracy for different score ranges point to serious limitations of interpretability of single indexes that are intended to represent classification accuracy. Suggestions are made for estimating classification accuracy for critical score ranges.