The validity and comparability of motivational constructs in TIMSS 2019

Abstract In this article, we examined the construct-related validity of the motivational measures included in TIMSS 2019 of Grade 8 in three Nordic countries— Finland, Sweden, and Norway. We utilized confirmatory factor analysis (CFA) to create three factors from the items of TIMSS Students’ Attitudes Toward Mathematics scales: (1) Students Confident in Mathematics, (2) Students Like Learning Mathematics, and (3) Students Value Mathematics. The model with three factors fitted well in the combined data from the three Nordic countries, but the association with mathematics achievement revealed a suppression effect and/or multicollinearity problems with the factors, which limits the interpretations of the results. Thus, we also tested an alternative model comprising three first-order factors and one second-order factor. We named this second-order factor “Motivation.” The association between the second-order factor and mathematics achievement was found to be strong and positive ( = 0.64, p < 0.001). In addition, we examined the measurement invariance of the second-order factor in the Nordic context with multigroup CFA. There was metric, but not scalar, invariance among Finland, Norway, and Sweden. This implies that the association between the factors and other variables can be compared among the three Nordic countries, but the mean level of motivation among these countries cannot be compared without caution. These findings emphasize the importance of acknowledging the validity and comparability of the measures utilized in conducting secondary analyses from data of international large-scale assessments to obtain interpretable results and meaningful conclusions.