Comparisons of Methodologies and Results in Vertical Scaling for Educational Achievement Tests

A number of vertical scaling methodologies were examined in this article. Scaling variations included data collection design, scaling method, item response theory (IRT) scoring procedure, and proficiency estimation method. Vertical scales were developed for Grade 3 through Grade 8 for 4 content areas and 9 simulated datasets. A total of 11 scaling variations were investigated for both real and simulated data. When the assumptions were met with the simulated data, all 11 scaling variations investigated were able to preserve the general characteristics of the scales. With the real data, vertical scales using all the methods showed decelerating growth from lower to higher grades. For within-grade variability, the Thurstone method produced scales with increasing variability over grades, whereas the IRT methods produced scales with fluctuating or decreasing variability over grades. Consequently, the growth patterns of high- and low-achieving students tended to differ across scaling methodologies. The scaling designs produced scales with dissimilar properties, especially for the tests that tended to be less homogeneous in content across grades and for tests that included testlet-based items. Discussion of the findings is provided, followed by a description of limitations of the study and possibilities for future research. Practical implications of the study also are discussed.