Stability of Country Rankings Across Item Formats in the Third International Mathematics and Science Study

Multiple‐choice, short‐answer, and extended‐response item formats were used in the Third International Mathematics and Science Study to assess student achievement in mathematics and science at Grades 7 and 8 in more than 40 countries around the world. Data pertaining to science indicate that the standings of some countries relative to others change when performance is measured via the different item formats. The question addressed in the present article is the following: Can the instability of ranks in this case be attributed principally to item format, or are other important factors at work? It is argued that the findings provide further evidence that comparing student achievement across countries is a very complex undertaking indeed.