Effects of the “highly adaptive testing design for PISA” on plausible value-based population estimates

Background The recently introduced highly adaptive testing (HAT) design allows for a transparent and straightforward construction of the testing design of the Programme for International Student Assessment (PISA). It outperformed the multistage testing (MST) design used in PISA 2018 regarding test information and RMSE of individual ability estimates, but it is yet unknown whether it can also promote the estimation of population statistics which are used for reporting PISA results. The present paper thus examines the impact of HAT on plausible value-based estimates of population statistics (country means, standard deviations, proportions of students on proficiency levels) and the error variance components associated with these estimates. Methods The study involved simulating the PISA 2018 Reading assessment under both the MST design used in PISA 2018 and the HAT design across 18 OECD countries ( N = 110,958 students), utilizing the publicly available assessment and background data, and the official PISA item pools for Reading, Mathematics, and Science. Data generation, item administration, and population modeling, including the drawing of the PVs, followed the official PISA 2018 procedures. Results The HAT design significantly reduced the mean imputation variance of the country-specific mean reading proficiency by 19%, compared to the MST design. With regard to the proportions on the proficiency levels, the HAT design led to significant reductions in the imputation variance (− 8 to − 13%) and the total error variance (up to − 9%) compared to the MST design. In most countries and on average, the proportions on the proficiency Levels V and VI were significantly smaller for the HAT design; this indicates an overestimation of these proportions with the MST design. Conclusions The HAT significantly increases the measurement precision of population statistics. Moreover, it seems to make a more nuanced categorizations of individuals into proficiency levels possible. These results on population level illustrate the importance of using the best possible design. This will become even more important if future PISA assessments utilize improved stratification schemes that reduce the sampling variance, as the influence of the design on the precision of the reported population statistics will increase.