Cross-national Comparability of Survey Attitude Measures

Many international surveys include batteries of questions which are combined to form scales by secondary analysts who often treat these scales as unproblematic. However, to be able to make valid cross-national comparisons of values on scales such as these, we need to be sure that the variations genuinely reflect differences in populations rather than being due to various forms of error. In order to illustrate the kinds of problems that can occur, and to explore ways of dealing with them, we examine items from the 2003 ISSP module on national identity. We show that differences between countries in response rates and other features of survey design are associated with the substantive outcomes and that ignoring these methodological problems will affect the validity of cross-national comparisons. We also examine whether the items can be assumed to have equivalence of meaning across the different countries in terms of their contributions to measuring two dimensions of national identity: civic and ethnic. We conclude that certain items do not seem to have the same meaning in all countries, thus jeopardizing the validity of cross-national comparisons using these scales. We suggest that the methods used and the lessons to be learnt can be applied to other items and cross-national datasets.