Introduction: During the COVID-19 pandemic, reliable data to inform public health policy-making was needed. Misclassification or bias in crisis needs assessments can lead to inadequate or misdirected resources. This study assessed differences in prevalence and risk factor associations for psychological well-being across two datasets examining the COVID-19 pandemic public health impact, focusing on how these differences relate to the sampling methods used. Methods: Data was obtained from two studies with different sampling Methods: 1) a quarterly cross-sectional panel study (N ~ 5.000 per wave), and 2) a biennial public health monitor with targeted social media sampling (N ~ 70.000 per wave). Both online questionnaires were distributed among Dutch youth (16-25 years) in June 2022, in which psychological well-being was assessed using the Mental Health Inventory 5 (MHI-5). Differences in the prevalence between the two data sets were assessed using Wilcoxon rank sum tests and Welch two-sample t-tests. Risk factors were studied using linear regression. To determine whether there was an inherent effect due to the difference in sampling methods, propensity score matching was conducted on demographics using nearest neighbor matching. Results: Psychological well-being was found to differ significantly between the two datasets, with the mean difference in MHI-5 score being 4 points lower for data obtained through social media sampling. Regarding risk factors, having trust in the future and experiencing various nonspecific symptoms revealed strong associations in both datasets. Between the datasets, notable associations were found with ongoing suffering from COVID-19-related experienced events in the panel data and with primarily demographic variables in the social media sampling data. The significantly lower MHI-5 score persisted after matching the participants based on demographics. Conclusion: Findings underscore the importance of taking precautions with interpreting the findings from social media-based sampling methods, as they run an increased risk of producing biased information for public health policy-making.
Rahmon et al. (Sun,) studied this question.