Hownotto reason with PISA data: an ironic investigation

AbstractInternational comparative testing, such as the OECD Programme for International Student Assessment (PISA), has considerable impact on policy-makers, the media and the general public. A central assumption underlying PISA is that global variation in students' academic performance is attributable to national educational structures and policies. The aim of this article is to demonstrate the problematic nature of this assumption. Rather than critiquing it from the outside, we turn the tools, data and presumptions of the current discourse of international comparisons upon themselves, showing that this assumption is refuted by analysis of immigrant student test scores. Data from the PISA 2009 tests show that Chinese immigrant students in New Zealand and Australia achieve math scores that are more similar to those of students in Shanghai than to their non-immigrant Australian and New Zealand peers. Thus, cultural background appears to be more consequential for the educational attainment of Chinese immigrant students than exposure to the educational systems of Australia or New Zealand. We discuss limitations of our analysis, using them as basis to discuss the shortcomings of PISA more generally.Keywords: comparativecritical analysis Notes1. Remarks made by Alison Wolf in symposium on How Countries Formulate Education Policies in Response to International Test-score Comparisons, AERA 2012 Annual Meeting, Vancouver, April 15, 2012.2. For the purpose of our exercise, we used the first plausible value in math (PV1MATH). The analysis was carried out using the student sample weights (W_FSTUWT). The weights were normalised in order to ensure that the sum of the weights is equal to the number of observations (see OECD Citation2009, 37).3. See Byun and Park (Citation2012) regarding the limited direct contribution of cram schooling to student achievement, and Ma, Jong, and Yuan (Citation2013) for an alternative explanation of East Asian students' success on PISA.4. This document is 'the result of a collaborative effort between the OECD, the NCEE in Washington, government officials of the case study countries discussed, as well as international experts with extensive expertise in analysing the performance of education systems internationally' (OECD Citation2011, 3).5. No countries share the precise profiles of the other leading systems (Korea, Finland, Hong Kong and Singapore), nor do they share characteristics with one another.