The Role of Item Format in the PISA 2018 Mathematics Literacy Assessment: A Cross-Country Study

When sources unrelated to the construct being measured affect the difficulty of assessment items, the validity of the assessment is compromised. Using the response data from over 260,000 students across 71 countries in the Programme for International Student Assessment (PISA) 2018 mathematical literacy assessment and cross-classified mixed effects models, we examined three validity concerns associated with the construct-irrelevant factor of item format. The first concern pertains to whether the format influenced item difficulty. The second explores if the impact of item format on difficulty varied across countries, potentially undermining PISA’s foundational goal of facilitating fair and meaningful country comparisons. The third concern investigates if item format effects differed between genders within the same country, possibly affecting assessment fairness. Our findings confirm all three concerns. Item format contributed to an average of 12% of variance in item difficulties across countries ---a substantial influence given its construct-irrelevant nature. Notably, the effect of item format was non-uniform across countries, with up to 30% of the variance in item difficulties being accounted for by format in lower-performing countries, compared to around 10% in higher-performing countries, challenging the comparability of global educational outcomes. The impact of gender on item format differences was present but minor and less concerning than the disparities observed across countries. We stress the need for assessment designers to carefully select and balance item formats to minimise their impact on difficulty, tailor assessments to diverse educational contexts, and continuously refine assessment design to uphold the validity and fairness of educational assessment globally.