Comparative Measures of Reliability for Integrated Datasets

Conference slides for the talk: Singh, R. K., and Repke, L. (2023). Comparative Measures of Reliability for Integrated Datasets.European Survey Research Association (ESRA) 2023 Conference, Università degli Studi di Milano-Bicocca, Milano, 2023-07-17. https://www.europeansurveyresearch.org/conf2023/prog.php?sess=76#418 Conference session abstract: Many research and infrastructure projects in the social sciences pool survey data from different sources, measured with different contexts, or fielded in different survey modes. Such projects often have to assess the comparability of different measurements before merging different source variables into a homogenous target variable.When researchers combine different variables into one, they often overlook one aspect of comparability: reliability. Ideally, different data sources should have comparable reliabilities; that is, they should be subject to comparable levels of random error. Establishing comparable reliabilities is challenging because many survey instruments have only one item, which makes estimating reliabilities more cumbersome than for multi-item instruments in psychometry.We propose two novel approaches to assessing the comparability of the reliability of two single-item instruments measuring political interest and compare them with test-retest reliability. The first approach relies on the open-access tool Survey Quality Predictor (SQP). Based on a meta-analysis of a large pool of methodological experiments, SQP predicts the measurement quality of instruments for continuous latent variables based on the characteristics of the instrument. This approach offers a cost-effective way for ex-post harmonization projects to assess the comparability of reliabilities.The second approach is what we term comparative attenuation. To assess if two instruments measure the same concept, it is often desirable to correlate both with other related concepts. We demonstrate that the same setup can be used to infer the relative reliability of the two instruments. If both instruments are conceptually comparable, but one is less reliable than the other, then a very specific pattern of intercorrelations with validation concepts arises.In an empirical proof-of-principle study, we apply both methods alongside a conventional but more costly test-retest reliability estimation and illustrate this with two different survey questions capturing political interest.