Cautions on OECD'S Recent Educational Survey (PISA)

A new survey of the educational attainments of 15-year-olds was undertaken by OECD in Spring 2000 (the 'PISA survey'). Surprisingly, British pupils appeared to perform in mathematics much better than in an IEA survey carried out only one year previously. This paper examines four main differences in the objectives and methods adopted in the two surveys. (a) Questions in the previous IEA survey were directed to the mastery of the school syllabus by the relevant age-groups, whereas PISA was ostensibly directed to so-called 'everyday life' problems--which provides less guidance for policy on schooling. (b) The IEA survey was based on samples of whole classes including, for example, older pupils who had entered school late, or had repeated a class: PISA excluded the latter pupils as it was based strictly on a 12-months' period of birth; issues of variability of pupils' attainments within a class--important for a class's teachability--cannot therefore be examined in this OECD survey. (c) England's response rate for schools was particularly low (60%, compared with 95% in leading European countries), raising serious doubts as to the inclusion of low-attaining schools. (d) The response rate of pupils (within participating schools in the PISA survey) was lower in England than in any other country, and lower than in the previous IEA survey, suggesting a greater upward bias in reported average scores. The paper concludes that it is difficult to draw valid conclusions for Britain from this survey and planned repeats should be postponed until the underlying methodological problems have been resolved.