The Effects of Age‐based and Grade‐based Sampling on the Relative Standing of Countries in International Comparative Studies of Student Achievement

The investigation reported in this article was prompted by discrepancies between the published outcomes from two international tests of science achievement: the Second International Assessment of Educational Progress (IAEP2) administered in 1991 and the Third International Mathematics and Science Study (TIMSS) administered in 1995. One finding was that while average science achievement for Irish 13 year‐olds was reported to be at the low end of the distribution for the 20 participating countries in IAEP2, it was around the middle of the distribution for the 40 or so countries that participated in TIMSS in the early grades of secondary schooling. Initial comparisons suggested that there were also inconsistencies in outcomes for some of the 11 other countries that participated in both surveys, e.g. France, Portugal, and Switzerland. Analyses described in the article reveal that when sampling/population definition differences between the two surveys are accounted for, science achievement in Ireland was not at the low level suggested by initial interpretations of the IAEP2 data but was closer to the levels reported in TIMSS. While the sampling issue did not fully account for discrepancies with respect to the IAEP2/TIMSS outcomes for some countries, it is argued that the findings outlined in this article have a number of implications for policy‐makers using data from future international comparative studies of student achievement.